[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:04:40 dist-test-slave-l73v krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:04:40 dist-test-slave-l73v krb5kdc[399](info): set up 2 sockets
Sep 26 01:04:40 dist-test-slave-l73v krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 01:04:41.255038 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:04:41 dist-test-slave-l73v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:04:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:04:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:04:41.378914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43237
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:41.464377 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:41.464627 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:41.464651 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:41.465916 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:04:41.465970 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:41.465983 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:41.465996 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:04:41.466007 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:04:41.467598 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:41.467857 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:41.468118 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:04:41.473904 412 server_base.cc:1034] running on GCE node
W20260926 01:04:41.473886 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:41.473922 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:41.473893 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:41.474473 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:41.474800 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:41.475976 412 hybrid_clock.cc:648] HybridClock initialized: now 1790384681475956 us; error 33 us; skew 500 ppm
Sep 26 01:04:41 dist-test-slave-l73v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:41.482976 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:04:41.483534 412 webserver.cc:460] Webserver started at http://127.0.98.62:36215/ using document root <none> and password file <none>
I20260926 01:04:41.483750 412 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:41.483832 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:41.483968 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:41.485159 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "8f9abfdbb503499ea80dd486516b9911"
format_stamp: "Formatted at 2026-09-26 01:04:41 on dist-test-slave-l73v"
server_key: "9da3c1ea514df5d3bf6fb0bb7e80580a"
server_key_iv: "4aa0b686816f0de4de13dfcca4d52ccf"
server_key_version: "encryptionkey@0"
I20260926 01:04:41.485550 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f9abfdbb503499ea80dd486516b9911"
format_stamp: "Formatted at 2026-09-26 01:04:41 on dist-test-slave-l73v"
server_key: "9da3c1ea514df5d3bf6fb0bb7e80580a"
server_key_iv: "4aa0b686816f0de4de13dfcca4d52ccf"
server_key_version: "encryptionkey@0"
I20260926 01:04:41.487061 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:04:41.487897 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:41.488094 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:04:41.488185 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "8f9abfdbb503499ea80dd486516b9911"
format_stamp: "Formatted at 2026-09-26 01:04:41 on dist-test-slave-l73v"
server_key: "9da3c1ea514df5d3bf6fb0bb7e80580a"
server_key_iv: "4aa0b686816f0de4de13dfcca4d52ccf"
server_key_version: "encryptionkey@0"
I20260926 01:04:41.488252 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:41.500188 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:41.502904 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:41.503087 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:41.507345 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43237
I20260926 01:04:41.507371 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43237 every 8 connection(s)
I20260926 01:04:41.507747 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:04:41.508667 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:41.511137 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911: Bootstrap starting.
I20260926 01:04:41.511723 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:41.511988 480 log.cc:824] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:41.512773 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911: No bootstrap required, opened a new log
I20260926 01:04:41.514192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 412
I20260926 01:04:41.514286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:04:41.514671 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9abfdbb503499ea80dd486516b9911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 01:04:41.514829 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:41.514851 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9abfdbb503499ea80dd486516b9911, State: Initialized, Role: FOLLOWER
I20260926 01:04:41.514988 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9abfdbb503499ea80dd486516b9911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 01:04:41.515060 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:04:41.515084 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:04:41.515093 392 external_mini_cluster.cc:1428] Setting key b789ebc07b67dff995459a9154aa7220
I20260926 01:04:41.515131 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:41.515805 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9abfdbb503499ea80dd486516b9911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 01:04:41.515928 480 leader_election.cc:302] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f9abfdbb503499ea80dd486516b9911; no voters:
I20260926 01:04:41.516173 480 leader_election.cc:288] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:04:41.516294 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:41.516544 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [term 1 LEADER]: Becoming Leader. State: Replica: 8f9abfdbb503499ea80dd486516b9911, State: Running, Role: LEADER
I20260926 01:04:41.516554 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:04:41.517421 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9abfdbb503499ea80dd486516b9911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 01:04:41.518098 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f9abfdbb503499ea80dd486516b9911. Latest consensus state: current_term: 1 leader_uuid: "8f9abfdbb503499ea80dd486516b9911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9abfdbb503499ea80dd486516b9911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } } }
I20260926 01:04:41.518193 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:41.518494 499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:04:41.518468 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f9abfdbb503499ea80dd486516b9911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9abfdbb503499ea80dd486516b9911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } } }
I20260926 01:04:41.518707 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:41.519148 499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:04:41.522917 499 catalog_manager.cc:1347] Generated new cluster ID: f7385a03f35c4cef8eb868ca9b6e8dd2
I20260926 01:04:41.522990 499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:04:41 dist-test-slave-l73v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:41.532821 499 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:04:41.533674 499 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:04:41.536266 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:41.516439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.0.98.62:43237)
0926 01:04:41.516717 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:41.516726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:41.516761 (+ 35us) server_negotiation.cc:407] Connection header received
0926 01:04:41.519486 (+ 2725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:41.519495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:41.519556 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:41.519714 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:41.522164 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:41.523653 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:41.525342 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:41.526029 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:41.526158 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:41.531548 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:41.531554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:41.531555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:41.531570 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:41.533819 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:41.534541 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:41.534545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:41.534545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:41.534675 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:41.534998 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:41.534999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:41.534999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:41.535308 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:41.535462 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:41.535842 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 01:04:41.536015 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":88,"threads_started":1}
I20260926 01:04:41.546617 499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f9abfdbb503499ea80dd486516b9911: Generated new TSK 0
I20260926 01:04:41.546823 499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:04:41.582077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:41.682972 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:41.683195 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:41.683228 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:41.684484 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:41.684531 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:41.684571 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:04:41.686146 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:41.686440 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:41.686661 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:41.692202 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:41.692296 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:41.692240 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:41.692364 506 server_base.cc:1034] running on GCE node
I20260926 01:04:41.692690 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:41.692946 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:41.694085 506 hybrid_clock.cc:648] HybridClock initialized: now 1790384681694075 us; error 30 us; skew 500 ppm
Sep 26 01:04:41 dist-test-slave-l73v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:41.696614 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:04:41.697104 506 webserver.cc:460] Webserver started at http://127.0.98.1:45287/ using document root <none> and password file <none>
I20260926 01:04:41.697322 506 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:41.697369 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:41.697464 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:41.698572 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "0575dc6bf5b24b20a3596373ea20bc34"
format_stamp: "Formatted at 2026-09-26 01:04:41 on dist-test-slave-l73v"
server_key: "264d012841b25da88b2d51c7468c1b12"
server_key_iv: "e3dfd356ad515378b1c4a0bd08a7742e"
server_key_version: "encryptionkey@0"
I20260926 01:04:41.698894 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0575dc6bf5b24b20a3596373ea20bc34"
format_stamp: "Formatted at 2026-09-26 01:04:41 on dist-test-slave-l73v"
server_key: "264d012841b25da88b2d51c7468c1b12"
server_key_iv: "e3dfd356ad515378b1c4a0bd08a7742e"
server_key_version: "encryptionkey@0"
I20260926 01:04:41.700189 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:41.700934 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:41.701128 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:41.701228 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "0575dc6bf5b24b20a3596373ea20bc34"
format_stamp: "Formatted at 2026-09-26 01:04:41 on dist-test-slave-l73v"
server_key: "264d012841b25da88b2d51c7468c1b12"
server_key_iv: "e3dfd356ad515378b1c4a0bd08a7742e"
server_key_version: "encryptionkey@0"
I20260926 01:04:41.701288 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:41.721659 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:41.724373 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:41.724514 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:41.724726 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:41.725085 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:41.725124 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:41.725154 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:41.725171 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:41.731298 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41555
I20260926 01:04:41.731346 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41555 every 8 connection(s)
I20260926 01:04:41.731675 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:04:41 dist-test-slave-l73v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:41.738008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 506
I20260926 01:04:41.738078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:04:41.738240 392 external_mini_cluster.cc:1428] Setting key 0c672b026b987782a1077bed6ca63138
I20260926 01:04:41.741449 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:41.732447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46593 (local address 127.0.98.62:43237)
0926 01:04:41.732525 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:41.732527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:41.732728 (+ 201us) server_negotiation.cc:407] Connection header received
0926 01:04:41.733262 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:41.733263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:41.733276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:41.733332 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:41.734314 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:41.734884 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:41.735892 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:41.736097 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:41.736155 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:41.738677 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:41.738679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:41.738679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:41.738686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:41.740032 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:41.740464 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:41.740465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:41.740465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:41.740528 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:41.740769 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:41.740770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:41.740770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:41.740895 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:41.740961 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:41.741247 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 01:04:41.741354 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:04:41.741706 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:41.732377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43237 (local address 127.0.98.1:46593)
0926 01:04:41.732602 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:41.732646 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:41.733147 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:41.733414 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:41.733418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:41.734002 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:41.734213 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:41.734219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:41.734978 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:41.734979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:41.735808 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:41.735810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:41.736171 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:41.736172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:41.736305 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:41.736845 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:41.736854 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:41.738576 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:41.740127 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:41.740138 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:41.740141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:41.740386 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:41.740585 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:41.740586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:41.740586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:41.740711 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:41.740953 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:41.740960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:41.741121 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 01:04:41.741217 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 01:04:41.741397 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260926 01:04:41.742331 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43237
I20260926 01:04:41.742519 635 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:41.742722 635 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 01:04:41.743255 444 ts_manager.cc:194] Registered new tserver with Master: 0575dc6bf5b24b20a3596373ea20bc34 (127.0.98.1:41555)
I20260926 01:04:41.743880 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46593
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:04:41.767385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:41.851560 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:41.851799 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:41.851827 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:41.853114 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:41.853173 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:41.853204 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:04:41.854797 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:41.855031 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:41.855237 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:41.861452 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:41.861384 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:41.861652 640 server_base.cc:1034] running on GCE node
W20260926 01:04:41.861871 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:41.862126 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:41.862409 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:41.863631 640 hybrid_clock.cc:648] HybridClock initialized: now 1790384681863620 us; error 34 us; skew 500 ppm
Sep 26 01:04:41 dist-test-slave-l73v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:41.866463 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:04:41.866974 640 webserver.cc:460] Webserver started at http://127.0.98.2:45679/ using document root <none> and password file <none>
I20260926 01:04:41.867189 640 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:41.867237 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:41.867353 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:41.868475 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "768dd9a3cc694e629a3cb1c30568bfd0"
format_stamp: "Formatted at 2026-09-26 01:04:41 on dist-test-slave-l73v"
server_key: "5663ad83d6f7466e1ba228612319c83f"
server_key_iv: "5a09ef77070627f9c8c3deb7e7865c9a"
server_key_version: "encryptionkey@0"
I20260926 01:04:41.868839 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "768dd9a3cc694e629a3cb1c30568bfd0"
format_stamp: "Formatted at 2026-09-26 01:04:41 on dist-test-slave-l73v"
server_key: "5663ad83d6f7466e1ba228612319c83f"
server_key_iv: "5a09ef77070627f9c8c3deb7e7865c9a"
server_key_version: "encryptionkey@0"
I20260926 01:04:41.870361 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:04:41.871441 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:41.871645 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:41.871737 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "768dd9a3cc694e629a3cb1c30568bfd0"
format_stamp: "Formatted at 2026-09-26 01:04:41 on dist-test-slave-l73v"
server_key: "5663ad83d6f7466e1ba228612319c83f"
server_key_iv: "5a09ef77070627f9c8c3deb7e7865c9a"
server_key_version: "encryptionkey@0"
I20260926 01:04:41.871816 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:41.888139 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:41.890735 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:41.890906 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:41.891134 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:41.891435 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:41.891469 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:41.891489 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:41.891515 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:41.898425 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36435
I20260926 01:04:41.898516 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36435 every 8 connection(s)
I20260926 01:04:41.898828 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:04:41.903000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 640
I20260926 01:04:41.903087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:04:41.903227 392 external_mini_cluster.cc:1428] Setting key 7c4987a9fcdd6c443188024b0933e215
Sep 26 01:04:41 dist-test-slave-l73v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:41.909981 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:41.900098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39761 (local address 127.0.98.62:43237)
0926 01:04:41.900153 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:41.900155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:41.900372 (+ 217us) server_negotiation.cc:407] Connection header received
0926 01:04:41.900917 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:41.900919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:41.900930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:41.900994 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:41.902139 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:41.902676 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:41.904036 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:41.904394 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:41.904474 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:41.906945 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:41.906947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:41.906948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:41.906956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:41.908536 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:41.908993 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:41.908995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:41.908996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:41.909055 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:41.909299 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:41.909300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:41.909300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:41.909427 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:41.909488 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:41.909784 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 01:04:41.909886 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:04:41.910176 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:41.900083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43237 (local address 127.0.98.2:39761)
0926 01:04:41.900236 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:41.900281 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:41.900771 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:41.901056 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:41.901060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:41.901650 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:41.902001 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:41.902012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:41.902780 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:41.902781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:41.903932 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:41.903936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:41.904492 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:41.904493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:41.904642 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:41.905235 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:41.905246 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:41.906807 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:41.908636 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:41.908646 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:41.908649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:41.908914 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:41.909113 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:41.909114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:41.909114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:41.909241 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:41.909487 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:41.909494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:41.909656 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 01:04:41.909752 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 01:04:41.909939 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260926 01:04:41.910696 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43237
I20260926 01:04:41.910799 769 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:41.910985 769 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:04:41.911450 444 ts_manager.cc:194] Registered new tserver with Master: 768dd9a3cc694e629a3cb1c30568bfd0 (127.0.98.2:36435)
I20260926 01:04:41.911892 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:04:41.934307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:42.017962 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:42.018179 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:42.018205 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:42.019585 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:42.019639 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:42.019670 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:04:42.021246 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:42.021513 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:42.021749 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:42.026811 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:42.026870 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:42.026942 774 server_base.cc:1034] running on GCE node
W20260926 01:04:42.026811 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:42.027231 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:42.027485 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:42.028661 774 hybrid_clock.cc:648] HybridClock initialized: now 1790384682028640 us; error 37 us; skew 500 ppm
Sep 26 01:04:42 dist-test-slave-l73v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:42.031298 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:04:42.031790 774 webserver.cc:460] Webserver started at http://127.0.98.3:45933/ using document root <none> and password file <none>
I20260926 01:04:42.032055 774 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:42.032106 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:42.032240 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:42.033340 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "0846f2983f124ef4b90fd960f8885c88"
format_stamp: "Formatted at 2026-09-26 01:04:42 on dist-test-slave-l73v"
server_key: "a0d55a31e6ef43b6661806ff891f9eef"
server_key_iv: "7829ef096fff46a7bc67d7f82ab3ec79"
server_key_version: "encryptionkey@0"
I20260926 01:04:42.033787 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0846f2983f124ef4b90fd960f8885c88"
format_stamp: "Formatted at 2026-09-26 01:04:42 on dist-test-slave-l73v"
server_key: "a0d55a31e6ef43b6661806ff891f9eef"
server_key_iv: "7829ef096fff46a7bc67d7f82ab3ec79"
server_key_version: "encryptionkey@0"
I20260926 01:04:42.035224 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:04:42.035996 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:42.036155 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:04:42.036242 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "0846f2983f124ef4b90fd960f8885c88"
format_stamp: "Formatted at 2026-09-26 01:04:42 on dist-test-slave-l73v"
server_key: "a0d55a31e6ef43b6661806ff891f9eef"
server_key_iv: "7829ef096fff46a7bc67d7f82ab3ec79"
server_key_version: "encryptionkey@0"
I20260926 01:04:42.036305 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:42.048213 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:42.050580 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:42.050732 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:42.050957 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:42.051265 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:42.051301 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:42.051332 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:42.051362 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:42.057847 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39011
I20260926 01:04:42.057914 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39011 every 8 connection(s)
I20260926 01:04:42.058209 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:04:42.059127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 774
I20260926 01:04:42.059198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:04:42.059325 392 external_mini_cluster.cc:1428] Setting key 8aff701bccc5699c4c322cd5a335b4c5
Sep 26 01:04:42 dist-test-slave-l73v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:42.068342 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:42.058977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60865 (local address 127.0.98.62:43237)
0926 01:04:42.059058 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:42.059061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:42.060247 (+ 1186us) server_negotiation.cc:407] Connection header received
0926 01:04:42.060817 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:42.060818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:42.060828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:42.060876 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:42.061829 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:42.062341 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:42.063423 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:42.063615 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:42.063676 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:42.065882 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:42.065883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:42.065884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:42.065892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:42.067149 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:42.067558 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:42.067558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:42.067559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:42.067619 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:42.067811 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:42.067811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:42.067811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:42.067925 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:42.067980 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:42.068200 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 01:04:42.068273 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 01:04:42.068560 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:42.058905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43237 (local address 127.0.98.3:60865)
0926 01:04:42.060105 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:42.060163 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:42.060723 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:42.060935 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:42.060938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:42.061505 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:42.061706 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:42.061711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:42.062504 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:42.062505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:42.063329 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:42.063332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:42.063683 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:42.063684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:42.063783 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:42.064297 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:42.064306 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:42.065792 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:42.067223 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:42.067231 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:42.067233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:42.067488 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:42.067673 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:42.067673 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:42.067673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:42.067769 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:42.067974 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:42.067977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:42.068100 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 01:04:42.068179 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 01:04:42.068305 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":83,"threads_started":1}
I20260926 01:04:42.069023 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43237
I20260926 01:04:42.069125 903 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:42.069315 903 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 01:04:42.069731 444 ts_manager.cc:194] Registered new tserver with Master: 0846f2983f124ef4b90fd960f8885c88 (127.0.98.3:39011)
I20260926 01:04:42.070083 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60865
I20260926 01:04:42.071321 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 01:04:42 dist-test-slave-l73v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384682, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 01:04:42 dist-test-slave-l73v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384682, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:04:42.094959 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:42.087080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.0.98.1:41555)
0926 01:04:42.087275 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:42.087279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:42.087291 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:04:42.087319 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:42.087321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:42.087335 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:42.087407 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:42.087956 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:42.088739 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:42.089483 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:42.089815 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:42.089878 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:42.092231 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:42.092239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:42.092240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:42.092247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:42.093795 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:42.094150 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:42.094153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:42.094154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:42.094218 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:42.094408 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:42.094409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:42.094409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:42.094562 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:42.094715 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:42.094760 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 01:04:42.094839 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
W20260926 01:04:42.095326 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58000
Sep 26 01:04:42 dist-test-slave-l73v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:04:42 dist-test-slave-l73v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:04:42.117444 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:42.110247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.0.98.1:41555)
0926 01:04:42.110313 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:42.110315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:42.110328 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:04:42.110428 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:42.110429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:42.110439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:42.110488 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:42.111055 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:42.111559 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:42.112359 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:42.112613 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:42.112680 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:42.115128 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:42.115130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:42.115130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:42.115139 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:42.116409 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:42.116754 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:42.116755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:42.116755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:42.116807 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:42.116975 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:42.116976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:42.116977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:42.117075 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:42.117182 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:42.117264 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 01:04:42.117347 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:04:42.118063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 506
I20260926 01:04:42.123268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 640
I20260926 01:04:42.128273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 774
I20260926 01:04:42.132308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 412
2026-09-26T01:04:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1857 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:04:42 dist-test-slave-l73v krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:04:42 dist-test-slave-l73v krb5kdc[917](info): set up 2 sockets
Sep 26 01:04:42 dist-test-slave-l73v krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 01:04:43.093869 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:04:43 dist-test-slave-l73v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:04:43.222177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38913
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:43.304335 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:43.304575 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:43.304610 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:43.305873 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:04:43.305930 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:43.305953 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:43.305967 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:04:43.305982 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:04:43.307658 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:43.307963 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:43.308192 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:43.313519 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:43.313587 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:43.313735 930 server_base.cc:1034] running on GCE node
W20260926 01:04:43.313678 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:43.314043 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:43.314352 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:43.315562 930 hybrid_clock.cc:648] HybridClock initialized: now 1790384683315539 us; error 39 us; skew 500 ppm
Sep 26 01:04:43 dist-test-slave-l73v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:43.319096 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:04:43.319636 930 webserver.cc:460] Webserver started at http://127.0.98.62:44989/ using document root <none> and password file <none>
I20260926 01:04:43.319864 930 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:43.319916 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:43.320048 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:43.321211 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "4a30689cd20a404aba34f1cce7ca46a6"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "4cc6d9fba4b64a6367c338e7b36466d5"
server_key_iv: "e05f8c9808caaf50e27f1440cb9b933d"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.321565 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a30689cd20a404aba34f1cce7ca46a6"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "4cc6d9fba4b64a6367c338e7b36466d5"
server_key_iv: "e05f8c9808caaf50e27f1440cb9b933d"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.323064 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:04:43.324034 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:43.324241 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:43.324342 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "4a30689cd20a404aba34f1cce7ca46a6"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "4cc6d9fba4b64a6367c338e7b36466d5"
server_key_iv: "e05f8c9808caaf50e27f1440cb9b933d"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.324412 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:43.338322 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:43.340610 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:43.340735 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:43.344480 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38913
I20260926 01:04:43.344533 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38913 every 8 connection(s)
I20260926 01:04:43.344859 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:04:43.345772 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:43.346943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 930
I20260926 01:04:43.347018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:04:43.347153 392 external_mini_cluster.cc:1428] Setting key 66ecf3d18e9c60494de912cd994e4cff
I20260926 01:04:43.348467 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6: Bootstrap starting.
I20260926 01:04:43.349081 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:43.349341 998 log.cc:824] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:43.350016 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6: No bootstrap required, opened a new log
I20260926 01:04:43.351910 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a30689cd20a404aba34f1cce7ca46a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38913 } }
I20260926 01:04:43.352103 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:43.352147 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a30689cd20a404aba34f1cce7ca46a6, State: Initialized, Role: FOLLOWER
I20260926 01:04:43.352303 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a30689cd20a404aba34f1cce7ca46a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38913 } }
I20260926 01:04:43.352381 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:04:43.352420 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:04:43.352468 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:43.353072 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a30689cd20a404aba34f1cce7ca46a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38913 } }
I20260926 01:04:43.353196 998 leader_election.cc:302] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a30689cd20a404aba34f1cce7ca46a6; no voters:
I20260926 01:04:43.353387 998 leader_election.cc:288] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:04:43.353428 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:43.353593 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [term 1 LEADER]: Becoming Leader. State: Replica: 4a30689cd20a404aba34f1cce7ca46a6, State: Running, Role: LEADER
Sep 26 01:04:43 dist-test-slave-l73v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:43.353649 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:04:43.353689 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a30689cd20a404aba34f1cce7ca46a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38913 } }
I20260926 01:04:43.354014 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a30689cd20a404aba34f1cce7ca46a6. Latest consensus state: current_term: 1 leader_uuid: "4a30689cd20a404aba34f1cce7ca46a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a30689cd20a404aba34f1cce7ca46a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38913 } } }
I20260926 01:04:43.354101 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:43.354022 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a30689cd20a404aba34f1cce7ca46a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a30689cd20a404aba34f1cce7ca46a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38913 } } }
I20260926 01:04:43.354203 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:43.354489 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:04:43.355008 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:04:43.358281 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.347676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.0.98.62:38913)
0926 01:04:43.347862 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:43.347870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:43.347896 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:04:43.348331 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:43.348338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:43.348388 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:43.348519 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:43.349193 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.350189 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.350969 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.351413 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.351512 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.354286 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:43.354292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:43.354292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:43.354308 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:43.356509 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:43.356900 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:43.356904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:43.356904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:43.357029 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:43.357266 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:43.357267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:43.357267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:43.357600 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:43.357745 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:43.357886 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 01:04:43.358038 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260926 01:04:43.359746 1009 catalog_manager.cc:1347] Generated new cluster ID: 6d8115cc0ec2439ebfe5ea4b5c5b5041
I20260926 01:04:43.359802 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:04:43.373584 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:04:43.374163 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:04:43.380236 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a30689cd20a404aba34f1cce7ca46a6: Generated new TSK 0
I20260926 01:04:43.380486 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:04:43.413249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38913
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:43.527297 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:43.527623 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:43.527671 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:43.529125 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:43.529186 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:43.529232 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:04:43.530956 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:43.531242 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:43.531486 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:43.536998 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:43.536999 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:43.536998 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:43.537251 1024 server_base.cc:1034] running on GCE node
I20260926 01:04:43.537473 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:43.537779 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:43.538930 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790384683538912 us; error 33 us; skew 500 ppm
Sep 26 01:04:43 dist-test-slave-l73v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:43.541536 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:04:43.541973 1024 webserver.cc:460] Webserver started at http://127.0.98.1:43953/ using document root <none> and password file <none>
I20260926 01:04:43.542160 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:43.542198 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:43.542294 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:43.543381 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "30732ed150214e11bb37804cabd1d497"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "bfeb8a6f1281cb942ec78a26ecd5bac5"
server_key_iv: "0a8bad8d486f43ed896dc28761dd1abc"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.543700 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "30732ed150214e11bb37804cabd1d497"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "bfeb8a6f1281cb942ec78a26ecd5bac5"
server_key_iv: "0a8bad8d486f43ed896dc28761dd1abc"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.544927 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:04:43.545753 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:43.545944 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:04:43.546033 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "30732ed150214e11bb37804cabd1d497"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "bfeb8a6f1281cb942ec78a26ecd5bac5"
server_key_iv: "0a8bad8d486f43ed896dc28761dd1abc"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.546100 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:43.560873 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:43.563177 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:43.563295 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:43.563505 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:43.563772 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:43.563802 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:43.563838 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:43.563853 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:43.569339 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46519
I20260926 01:04:43.569389 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46519 every 8 connection(s)
I20260926 01:04:43.569715 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:04:43 dist-test-slave-l73v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:43.579289 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.570588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39633 (local address 127.0.98.62:38913)
0926 01:04:43.570634 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:43.570636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:43.570802 (+ 166us) server_negotiation.cc:407] Connection header received
0926 01:04:43.571296 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:43.571297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:43.571307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:43.571369 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:43.572376 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.572922 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.574022 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.574245 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.574307 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.576504 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:43.576538 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:43.576538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:43.576546 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:43.577893 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:43.578292 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:43.578293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:43.578293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:43.578347 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:43.578625 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:43.578626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:43.578626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:43.578752 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:43.578816 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:43.579133 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 01:04:43.579217 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:04:43.579545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 1024
I20260926 01:04:43.579625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:04:43.579779 392 external_mini_cluster.cc:1428] Setting key 95c1a04538abe1be04eda00cc6ff90ef
I20260926 01:04:43.580006 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.570485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38913 (local address 127.0.98.1:39633)
0926 01:04:43.570654 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:43.570723 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:43.571196 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:43.571450 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:43.571453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:43.572044 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:43.572269 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.572274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.573036 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.573037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.573923 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.573925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.574314 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.574315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.574451 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.574988 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:43.574997 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:43.576392 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:43.577983 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:43.577990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:43.577992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:43.578227 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:43.578440 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:43.578441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:43.578442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:43.578546 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:43.578832 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:43.578839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:43.578977 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 01:04:43.579076 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 01:04:43.579220 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260926 01:04:43.580617 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38913
I20260926 01:04:43.580724 1153 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:43.580907 1153 heartbeater.cc:511] Master 127.0.98.62:38913 requested a full tablet report, sending...
I20260926 01:04:43.581544 961 ts_manager.cc:194] Registered new tserver with Master: 30732ed150214e11bb37804cabd1d497 (127.0.98.1:46519)
I20260926 01:04:43.582334 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39633
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:04:43.606588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38913
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:43.689255 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:43.689497 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:43.689525 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:43.691313 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:43.691381 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:43.691414 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:04:43.693177 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:43.693416 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:43.693629 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:43.698843 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:43.698837 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:43.698837 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:43.699319 1158 server_base.cc:1034] running on GCE node
I20260926 01:04:43.699506 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:43.699728 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:43.700892 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790384683700864 us; error 44 us; skew 500 ppm
Sep 26 01:04:43 dist-test-slave-l73v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:43.703487 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:04:43.703989 1158 webserver.cc:460] Webserver started at http://127.0.98.2:41607/ using document root <none> and password file <none>
I20260926 01:04:43.704211 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:43.704262 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:43.704391 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:43.705494 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "db77c886a5c94261b9117a5e104aa52e"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "999e1677a2cec4737dac8a0da25b1905"
server_key_iv: "cd622f1ae8e09d5c2610f1330f076eee"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.705883 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "db77c886a5c94261b9117a5e104aa52e"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "999e1677a2cec4737dac8a0da25b1905"
server_key_iv: "cd622f1ae8e09d5c2610f1330f076eee"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.707353 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:04:43.708227 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:43.708433 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:04:43.708523 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "db77c886a5c94261b9117a5e104aa52e"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "999e1677a2cec4737dac8a0da25b1905"
server_key_iv: "cd622f1ae8e09d5c2610f1330f076eee"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.708623 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:43.723280 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:43.725674 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:43.725871 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:43.726106 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:43.726490 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:43.726531 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:43.726562 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:43.726586 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:43.732473 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34885
I20260926 01:04:43.732558 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34885 every 8 connection(s)
I20260926 01:04:43.732875 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:04:43 dist-test-slave-l73v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:43.741777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 1158
I20260926 01:04:43.741864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:04:43.742015 392 external_mini_cluster.cc:1428] Setting key b3b43c5d88e4ee595786a0278871332f
I20260926 01:04:43.742702 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.733688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40263 (local address 127.0.98.62:38913)
0926 01:04:43.733759 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:43.733760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:43.733952 (+ 192us) server_negotiation.cc:407] Connection header received
0926 01:04:43.734449 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:43.734450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:43.734459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:43.734511 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:43.735478 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.735977 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.737027 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.737233 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.737295 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.739677 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:43.739679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:43.739679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:43.739687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:43.741130 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:43.741538 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:43.741539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:43.741539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:43.741590 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:43.741901 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:43.741902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:43.741903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:43.742056 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:43.742131 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:43.742511 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 01:04:43.742603 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:04:43.742837 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.733626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38913 (local address 127.0.98.2:40263)
0926 01:04:43.733813 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:43.733857 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:43.734334 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:43.734578 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:43.734582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:43.735157 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:43.735361 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.735366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.736075 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.736076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.736934 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.736937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.737332 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.737333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.737465 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.738071 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:43.738086 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:43.739586 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:43.741217 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:43.741224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:43.741227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:43.741476 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:43.741671 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:43.741672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:43.741672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:43.741795 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:43.742126 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:43.742134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:43.742315 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 01:04:43.742470 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 01:04:43.742622 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260926 01:04:43.743448 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38913
I20260926 01:04:43.743561 1287 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:43.743760 1287 heartbeater.cc:511] Master 127.0.98.62:38913 requested a full tablet report, sending...
I20260926 01:04:43.744159 961 ts_manager.cc:194] Registered new tserver with Master: db77c886a5c94261b9117a5e104aa52e (127.0.98.2:34885)
I20260926 01:04:43.744642 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40263
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:04:43.770432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38913
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:43.852782 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:43.853020 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:43.853047 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:43.854486 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:43.854553 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:43.854598 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:04:43.856374 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:43.856606 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:43.856799 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:43.862023 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:43.862157 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:43.862141 1292 server_base.cc:1034] running on GCE node
W20260926 01:04:43.862044 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:43.862496 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:43.862771 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:43.863941 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790384683863929 us; error 33 us; skew 500 ppm
Sep 26 01:04:43 dist-test-slave-l73v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:43.866634 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:04:43.867122 1292 webserver.cc:460] Webserver started at http://127.0.98.3:38425/ using document root <none> and password file <none>
I20260926 01:04:43.867328 1292 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:43.867375 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:43.867501 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:43.868551 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "0e85d2c0c1834119a912915e7483a99c"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "a2f7667ad1fc1efa5dea49a6266bb592"
server_key_iv: "7e9ab73dd9c579d28e87b5e0f5d9ddca"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.868887 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0e85d2c0c1834119a912915e7483a99c"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "a2f7667ad1fc1efa5dea49a6266bb592"
server_key_iv: "7e9ab73dd9c579d28e87b5e0f5d9ddca"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.870247 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 01:04:43.871093 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:43.871308 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:43.871402 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "0e85d2c0c1834119a912915e7483a99c"
format_stamp: "Formatted at 2026-09-26 01:04:43 on dist-test-slave-l73v"
server_key: "a2f7667ad1fc1efa5dea49a6266bb592"
server_key_iv: "7e9ab73dd9c579d28e87b5e0f5d9ddca"
server_key_version: "encryptionkey@0"
I20260926 01:04:43.871472 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:43.882907 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:43.885226 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:43.885370 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:43.885596 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:43.885896 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:43.885929 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:43.885958 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:43.885982 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:43.892016 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46813
I20260926 01:04:43.892063 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46813 every 8 connection(s)
I20260926 01:04:43.892405 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:04:43.895263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 1292
I20260926 01:04:43.895349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:04:43.895474 392 external_mini_cluster.cc:1428] Setting key 88dd4c50fbd634d077c0638c0c419fb8
Sep 26 01:04:43 dist-test-slave-l73v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:43.901777 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.893119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47901 (local address 127.0.98.62:38913)
0926 01:04:43.893178 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:43.893180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:43.893402 (+ 222us) server_negotiation.cc:407] Connection header received
0926 01:04:43.893868 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:43.893869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:43.893879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:43.893950 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:43.894950 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.895619 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.896685 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.896887 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.896954 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.899282 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:43.899284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:43.899284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:43.899291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:43.900579 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:43.900925 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:43.900927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:43.900927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:43.900985 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:43.901213 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:43.901214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:43.901215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:43.901339 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:43.901402 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:43.901617 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 01:04:43.901710 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:04:43.901924 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.893086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38913 (local address 127.0.98.3:47901)
0926 01:04:43.893253 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:43.893296 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:43.893768 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:43.894015 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:43.894019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:43.894644 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:43.894850 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.894855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.895751 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.895752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.896588 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.896590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.897042 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.897043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.897182 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.897710 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:43.897719 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:43.899167 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:43.900641 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:43.900645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:43.900647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:43.900849 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:43.901043 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:43.901044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:43.901044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:43.901137 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:43.901396 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:43.901399 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:43.901519 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 01:04:43.901597 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:04:43.901721 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260926 01:04:43.902367 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38913
I20260926 01:04:43.902489 1421 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:43.902661 1421 heartbeater.cc:511] Master 127.0.98.62:38913 requested a full tablet report, sending...
I20260926 01:04:43.903008 961 ts_manager.cc:194] Registered new tserver with Master: 0e85d2c0c1834119a912915e7483a99c (127.0.98.3:46813)
I20260926 01:04:43.903393 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47901
I20260926 01:04:43.907073 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:04:43.914361 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.908536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.0.98.62:38913)
0926 01:04:43.908589 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:43.908591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:43.908648 (+ 57us) server_negotiation.cc:407] Connection header received
0926 01:04:43.908723 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:43.908724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:43.908734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:43.908774 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:43.909296 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.909778 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.910517 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.910719 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.910770 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.911951 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:43.911953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:43.911953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:43.911961 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:43.913321 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:43.913631 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:43.913633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:43.913633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:43.913689 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:43.913848 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:43.913849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:43.913849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:43.914002 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:43.914088 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:43.914195 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 01:04:43.914271 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:04:43.916563 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:04:43.918044 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:04:43.924295 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.920167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46813 (local address 127.0.0.1:43970)
0926 01:04:43.920348 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:43.920391 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:43.920457 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:43.920709 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:43.920713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:43.920715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:43.920790 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.920793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.922134 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.922135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.923200 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.923203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.924012 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.924013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.924081 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.924093 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 01:04:43.924176 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 01:04:43.924224 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":85,"thread_start_us":56,"threads_started":1}
I20260926 01:04:43.924326 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.920256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.98.3:46813)
0926 01:04:43.920447 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:43.920452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:43.920464 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:04:43.920596 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:43.920600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:43.920617 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:43.920688 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:43.920876 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.922030 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.923264 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.923900 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.923977 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.924036 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:43.924200 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 01:04:43.924245 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260926 01:04:43.924765 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.920359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34885 (local address 127.0.0.1:42440)
0926 01:04:43.920797 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:43.920805 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:43.920851 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:43.921101 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:43.921102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:43.921104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:43.921146 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.921147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.922066 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.922067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.923481 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.923484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.924528 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.924529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.924592 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.924597 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:04:43.924651 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 01:04:43.924695 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260926 01:04:43.925038 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.920567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.0.98.2:34885)
0926 01:04:43.920753 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:43.920758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:43.920899 (+ 141us) server_negotiation.cc:407] Connection header received
0926 01:04:43.920942 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:43.920945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:43.920959 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:43.921024 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:43.921218 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.921952 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.923569 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.924276 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.924349 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.924405 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:43.924886 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 01:04:43.924943 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260926 01:04:43.925058 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 97683b197eab46c8afed3eab6dd0d3b0 (DEFAULT_TABLE table=test-table [id=96b5fbfe1fc34ed7a344c12e365efc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:43.925452 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97683b197eab46c8afed3eab6dd0d3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:43.925531 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.920291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.0.98.1:46519)
0926 01:04:43.920543 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:43.920547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:43.920605 (+ 58us) server_negotiation.cc:407] Connection header received
0926 01:04:43.920750 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:43.920753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:43.920770 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:43.920842 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:43.920966 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.922039 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.924447 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.925171 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.925246 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.925302 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:43.925415 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 01:04:43.925458 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":76,"threads_started":1}
I20260926 01:04:43.925537 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:43.920173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46519 (local address 127.0.0.1:37720)
0926 01:04:43.920555 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:43.920563 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:43.920617 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:43.920844 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:43.920845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:43.920847 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:43.920901 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.920902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.922258 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.922258 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.923485 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:43.923487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:43.925261 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:43.925262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:43.925327 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:43.925332 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:04:43.925397 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 01:04:43.925438 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":46,"thread_start_us":86,"threads_started":1}
I20260926 01:04:43.926142 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 97683b197eab46c8afed3eab6dd0d3b0 (DEFAULT_TABLE table=test-table [id=96b5fbfe1fc34ed7a344c12e365efc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:43.926470 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97683b197eab46c8afed3eab6dd0d3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:43.926836 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 97683b197eab46c8afed3eab6dd0d3b0 (DEFAULT_TABLE table=test-table [id=96b5fbfe1fc34ed7a344c12e365efc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:43.927184 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97683b197eab46c8afed3eab6dd0d3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:43.928691 1438 tablet_bootstrap.cc:492] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c: Bootstrap starting.
I20260926 01:04:43.929327 1439 tablet_bootstrap.cc:492] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497: Bootstrap starting.
I20260926 01:04:43.929524 1438 tablet_bootstrap.cc:654] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:43.929965 1439 tablet_bootstrap.cc:654] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:43.930063 1438 log.cc:824] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:43.930281 1440 tablet_bootstrap.cc:492] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e: Bootstrap starting.
I20260926 01:04:43.930924 1440 tablet_bootstrap.cc:654] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:43.931039 1439 log.cc:824] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:43.931494 1438 tablet_bootstrap.cc:492] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c: No bootstrap required, opened a new log
I20260926 01:04:43.931470 1440 log.cc:824] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:43.931618 1438 ts_tablet_manager.cc:1397] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260926 01:04:43.931651 1439 tablet_bootstrap.cc:492] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497: No bootstrap required, opened a new log
I20260926 01:04:43.931696 1439 ts_tablet_manager.cc:1397] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:04:43.932225 1440 tablet_bootstrap.cc:492] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e: No bootstrap required, opened a new log
I20260926 01:04:43.932317 1440 ts_tablet_manager.cc:1397] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:04:43.933156 1439 raft_consensus.cc:348] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30732ed150214e11bb37804cabd1d497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46519 } } peers { permanent_uuid: "0e85d2c0c1834119a912915e7483a99c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46813 } } peers { permanent_uuid: "db77c886a5c94261b9117a5e104aa52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } }
I20260926 01:04:43.933310 1439 raft_consensus.cc:374] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:43.933341 1439 raft_consensus.cc:729] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30732ed150214e11bb37804cabd1d497, State: Initialized, Role: FOLLOWER
I20260926 01:04:43.933477 1439 consensus_queue.cc:260] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30732ed150214e11bb37804cabd1d497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46519 } } peers { permanent_uuid: "0e85d2c0c1834119a912915e7483a99c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46813 } } peers { permanent_uuid: "db77c886a5c94261b9117a5e104aa52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } }
I20260926 01:04:43.933807 1439 ts_tablet_manager.cc:1428] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:04:43.933925 1153 heartbeater.cc:503] Master 127.0.98.62:38913 was elected leader, sending a full tablet report...
I20260926 01:04:43.933948 1438 raft_consensus.cc:348] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30732ed150214e11bb37804cabd1d497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46519 } } peers { permanent_uuid: "0e85d2c0c1834119a912915e7483a99c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46813 } } peers { permanent_uuid: "db77c886a5c94261b9117a5e104aa52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } }
I20260926 01:04:43.934149 1438 raft_consensus.cc:374] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:43.934186 1438 raft_consensus.cc:729] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e85d2c0c1834119a912915e7483a99c, State: Initialized, Role: FOLLOWER
I20260926 01:04:43.934297 1438 consensus_queue.cc:260] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30732ed150214e11bb37804cabd1d497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46519 } } peers { permanent_uuid: "0e85d2c0c1834119a912915e7483a99c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46813 } } peers { permanent_uuid: "db77c886a5c94261b9117a5e104aa52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } }
I20260926 01:04:43.934296 1440 raft_consensus.cc:348] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30732ed150214e11bb37804cabd1d497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46519 } } peers { permanent_uuid: "0e85d2c0c1834119a912915e7483a99c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46813 } } peers { permanent_uuid: "db77c886a5c94261b9117a5e104aa52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } }
I20260926 01:04:43.934505 1440 raft_consensus.cc:374] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:43.934556 1440 raft_consensus.cc:729] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db77c886a5c94261b9117a5e104aa52e, State: Initialized, Role: FOLLOWER
I20260926 01:04:43.934616 1438 ts_tablet_manager.cc:1428] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:04:43.934726 1421 heartbeater.cc:503] Master 127.0.98.62:38913 was elected leader, sending a full tablet report...
I20260926 01:04:43.934695 1440 consensus_queue.cc:260] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30732ed150214e11bb37804cabd1d497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46519 } } peers { permanent_uuid: "0e85d2c0c1834119a912915e7483a99c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46813 } } peers { permanent_uuid: "db77c886a5c94261b9117a5e104aa52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } }
I20260926 01:04:43.934973 1440 ts_tablet_manager.cc:1428] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:04:43.935102 1287 heartbeater.cc:503] Master 127.0.98.62:38913 was elected leader, sending a full tablet report...
W20260926 01:04:43.983479 1288 tablet.cc:2369] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:04:44.070425 1154 tablet.cc:2369] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:04:44.110556 1446 raft_consensus.cc:482] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:04:44.110710 1446 raft_consensus.cc:504] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30732ed150214e11bb37804cabd1d497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46519 } } peers { permanent_uuid: "0e85d2c0c1834119a912915e7483a99c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46813 } } peers { permanent_uuid: "db77c886a5c94261b9117a5e104aa52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } }
I20260926 01:04:44.111071 1446 leader_election.cc:288] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30732ed150214e11bb37804cabd1d497 (127.0.98.1:46519), 0e85d2c0c1834119a912915e7483a99c (127.0.98.3:46813)
I20260926 01:04:44.114801 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:44.111342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36039 (local address 127.0.98.3:46813)
0926 01:04:44.111393 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:44.111395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:44.111407 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:04:44.111435 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:44.111436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:44.111446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:44.111510 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:44.111666 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.112441 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.113850 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.114452 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.114531 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:44.114541 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:44.114687 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 01:04:44.114732 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:04:44.114826 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:44.111331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47715 (local address 127.0.98.1:46519)
0926 01:04:44.111376 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:44.111378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:44.111579 (+ 201us) server_negotiation.cc:407] Connection header received
0926 01:04:44.111612 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:44.111613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:44.111626 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:44.111677 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:44.111826 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.112547 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.113852 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.114468 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.114531 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:44.114541 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:44.114731 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 01:04:44.114772 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:04:44.114820 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:44.111273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46813 (local address 127.0.98.2:36039)
0926 01:04:44.111326 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:44.111338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:44.111385 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:44.111524 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:44.111525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:44.111527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:44.111601 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:44.111603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.112510 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.112511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:44.113739 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:44.113742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.114528 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.114529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:44.114627 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:44.114629 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:04:44.114679 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:04:44.114727 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":2}
I20260926 01:04:44.114827 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:44.111281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46519 (local address 127.0.98.2:47715)
0926 01:04:44.111513 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:44.111521 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:44.111571 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:44.111718 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:44.111719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:44.111720 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:44.111768 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:44.111769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.112623 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.112624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:44.113765 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:44.113766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.114528 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.114529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:44.114622 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:44.114626 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:04:44.114719 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 01:04:44.114744 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":78,"thread_start_us":96,"threads_started":1}
I20260926 01:04:44.115186 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97683b197eab46c8afed3eab6dd0d3b0" candidate_uuid: "db77c886a5c94261b9117a5e104aa52e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e85d2c0c1834119a912915e7483a99c" is_pre_election: true
I20260926 01:04:44.115315 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97683b197eab46c8afed3eab6dd0d3b0" candidate_uuid: "db77c886a5c94261b9117a5e104aa52e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30732ed150214e11bb37804cabd1d497" is_pre_election: true
I20260926 01:04:44.115371 1375 raft_consensus.cc:2454] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db77c886a5c94261b9117a5e104aa52e in term 0.
I20260926 01:04:44.115442 1107 raft_consensus.cc:2454] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db77c886a5c94261b9117a5e104aa52e in term 0.
I20260926 01:04:44.115645 1177 leader_election.cc:302] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e85d2c0c1834119a912915e7483a99c, db77c886a5c94261b9117a5e104aa52e; no voters:
I20260926 01:04:44.115789 1446 raft_consensus.cc:2781] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:04:44.115861 1446 raft_consensus.cc:482] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:04:44.115880 1446 raft_consensus.cc:3037] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:44.117000 1446 raft_consensus.cc:504] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30732ed150214e11bb37804cabd1d497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46519 } } peers { permanent_uuid: "0e85d2c0c1834119a912915e7483a99c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46813 } } peers { permanent_uuid: "db77c886a5c94261b9117a5e104aa52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } }
I20260926 01:04:44.117146 1446 leader_election.cc:288] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [CANDIDATE]: Term 1 election: Requested vote from peers 30732ed150214e11bb37804cabd1d497 (127.0.98.1:46519), 0e85d2c0c1834119a912915e7483a99c (127.0.98.3:46813)
I20260926 01:04:44.117338 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97683b197eab46c8afed3eab6dd0d3b0" candidate_uuid: "db77c886a5c94261b9117a5e104aa52e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30732ed150214e11bb37804cabd1d497"
I20260926 01:04:44.117425 1107 raft_consensus.cc:3037] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:44.118116 1107 raft_consensus.cc:2454] T 97683b197eab46c8afed3eab6dd0d3b0 P 30732ed150214e11bb37804cabd1d497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db77c886a5c94261b9117a5e104aa52e in term 1.
I20260926 01:04:44.118230 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97683b197eab46c8afed3eab6dd0d3b0" candidate_uuid: "db77c886a5c94261b9117a5e104aa52e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e85d2c0c1834119a912915e7483a99c"
I20260926 01:04:44.118320 1375 raft_consensus.cc:3037] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:44.118331 1176 leader_election.cc:302] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30732ed150214e11bb37804cabd1d497, db77c886a5c94261b9117a5e104aa52e; no voters:
I20260926 01:04:44.118443 1446 raft_consensus.cc:2781] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:44.118605 1446 raft_consensus.cc:686] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [term 1 LEADER]: Becoming Leader. State: Replica: db77c886a5c94261b9117a5e104aa52e, State: Running, Role: LEADER
I20260926 01:04:44.118711 1446 consensus_queue.cc:237] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30732ed150214e11bb37804cabd1d497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46519 } } peers { permanent_uuid: "0e85d2c0c1834119a912915e7483a99c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46813 } } peers { permanent_uuid: "db77c886a5c94261b9117a5e104aa52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } }
I20260926 01:04:44.119110 1375 raft_consensus.cc:2454] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db77c886a5c94261b9117a5e104aa52e in term 1.
I20260926 01:04:44.119570 962 catalog_manager.cc:5495] T 97683b197eab46c8afed3eab6dd0d3b0 P db77c886a5c94261b9117a5e104aa52e reported cstate change: term changed from 0 to 1, leader changed from <none> to db77c886a5c94261b9117a5e104aa52e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "db77c886a5c94261b9117a5e104aa52e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30732ed150214e11bb37804cabd1d497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e85d2c0c1834119a912915e7483a99c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db77c886a5c94261b9117a5e104aa52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } health_report { overall_health: HEALTHY } } }
W20260926 01:04:44.142987 1422 tablet.cc:2369] T 97683b197eab46c8afed3eab6dd0d3b0 P 0e85d2c0c1834119a912915e7483a99c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 01:04:44 dist-test-slave-l73v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:04:44.161950 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:44.153910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.0.98.1:46519)
0926 01:04:44.153970 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:44.153972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:44.153981 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:04:44.154042 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:44.154043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:44.154053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:44.154101 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:44.154721 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.155288 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.156080 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.156272 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.156331 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:44.158997 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:44.159012 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:44.159012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:44.159020 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:44.160647 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:44.161147 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:44.161151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:44.161152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:44.161221 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:44.161403 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:44.161405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:44.161405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:44.161580 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:44.161743 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:44.161786 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 01:04:44.161857 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 01:04:44 dist-test-slave-l73v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 01:04:44 dist-test-slave-l73v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:04:44.184751 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:44.177654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.0.98.1:46519)
0926 01:04:44.177752 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:44.177754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:44.177768 (+ 14us) server_negotiation.cc:407] Connection header received
0926 01:04:44.177855 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:44.177856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:44.177866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:44.177915 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:44.178527 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.179011 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.179723 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.179911 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.179965 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:44.182142 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:44.182144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:44.182144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:44.182153 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:44.183634 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:44.183991 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:44.183992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:44.183993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:44.184046 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:44.184206 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:44.184207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:44.184207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:44.184320 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:44.184411 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:44.184576 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 01:04:44.184659 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 01:04:44 dist-test-slave-l73v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:04:44 dist-test-slave-l73v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:04:44.207387 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:44.200097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.0.98.1:46519)
0926 01:04:44.200154 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:44.200156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:44.200204 (+ 48us) server_negotiation.cc:407] Connection header received
0926 01:04:44.200261 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:44.200262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:44.200272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:44.200319 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:44.201033 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.201579 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.202418 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:44.202623 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:44.202679 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:44.205010 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:44.205012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:44.205012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:44.205019 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:44.206273 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:44.206712 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:44.206713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:44.206713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:44.206777 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:44.206961 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:44.206962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:44.206962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:44.207058 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:44.207151 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:44.207251 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 01:04:44.207317 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 01:04:44.207881 1067 tablet_service.cc:2908] Rejecting scan request for tablet 97683b197eab46c8afed3eab6dd0d3b0: Uninitialized: clean time has not yet been initialized
I20260926 01:04:44.208640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 1024
I20260926 01:04:44.213201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 1158
I20260926 01:04:44.217402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 1292
I20260926 01:04:44.221339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 930
2026-09-26T01:04:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2089 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:04:44 dist-test-slave-l73v krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:04:44 dist-test-slave-l73v krb5kdc[1462](info): set up 2 sockets
Sep 26 01:04:44 dist-test-slave-l73v krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260926 01:04:45.293166 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 01:04:45.468897 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.176s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 01:04:45.653318 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.184s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 01:04:45.907112 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.254s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/test-user.keytab.
W20260926 01:04:46.462685 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.555s user 0.000s sys 0.002s
Sep 26 01:04:46 dist-test-slave-l73v krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:04:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260926 01:04:46.959302 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.475s user 0.001s sys 0.001s
I20260926 01:04:46.959643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:47.040072 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:47.040303 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:47.040326 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:47.041513 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:04:47.041563 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 01:04:47.041580 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:47.041599 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:47.041615 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:04:47.041630 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:04:47.043220 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39421
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:47.043434 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:47.043618 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:04:47.048632 1475 server_base.cc:1034] running on GCE node
W20260926 01:04:47.048568 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:47.048596 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:47.048604 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:47.049065 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:47.049391 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:47.050532 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790384687050515 us; error 36 us; skew 500 ppm
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:47.057894 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:04:47.058436 1475 webserver.cc:460] Webserver started at http://127.0.98.62:45765/ using document root <none> and password file <none>
I20260926 01:04:47.058678 1475 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:47.058732 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:47.058866 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:47.059953 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "7dcf7ef7565040bd8b1930fcafa01379"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "2861b32fb82d467d83160cc7c6bef2e1"
server_key_iv: "04acb933f78fc68f982189dbbc566561"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.060295 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "7dcf7ef7565040bd8b1930fcafa01379"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "2861b32fb82d467d83160cc7c6bef2e1"
server_key_iv: "04acb933f78fc68f982189dbbc566561"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.061626 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:04:47.062460 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.062668 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:47.062762 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "7dcf7ef7565040bd8b1930fcafa01379"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "2861b32fb82d467d83160cc7c6bef2e1"
server_key_iv: "04acb933f78fc68f982189dbbc566561"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.062821 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:47.082638 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:47.084833 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:47.084955 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:47.089169 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39421
I20260926 01:04:47.089236 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39421 every 8 connection(s)
I20260926 01:04:47.089535 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:04:47.090212 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:47.092557 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379: Bootstrap starting.
I20260926 01:04:47.093219 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:47.093495 1553 log.cc:824] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:47.094241 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379: No bootstrap required, opened a new log
I20260926 01:04:47.094705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 1475
I20260926 01:04:47.094792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:04:47.094954 392 external_mini_cluster.cc:1428] Setting key 024b990592076c57a93c26edec94d8cb
I20260926 01:04:47.096733 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcf7ef7565040bd8b1930fcafa01379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39421 } }
I20260926 01:04:47.096917 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:47.096952 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcf7ef7565040bd8b1930fcafa01379, State: Initialized, Role: FOLLOWER
I20260926 01:04:47.097122 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcf7ef7565040bd8b1930fcafa01379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39421 } }
I20260926 01:04:47.097210 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:04:47.097236 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:04:47.097283 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:47.097976 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcf7ef7565040bd8b1930fcafa01379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39421 } }
I20260926 01:04:47.098095 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dcf7ef7565040bd8b1930fcafa01379; no voters:
I20260926 01:04:47.098248 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:04:47.098294 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:47.098498 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [term 1 LEADER]: Becoming Leader. State: Replica: 7dcf7ef7565040bd8b1930fcafa01379, State: Running, Role: LEADER
I20260926 01:04:47.098520 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:04:47.098644 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcf7ef7565040bd8b1930fcafa01379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39421 } }
I20260926 01:04:47.099095 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dcf7ef7565040bd8b1930fcafa01379. Latest consensus state: current_term: 1 leader_uuid: "7dcf7ef7565040bd8b1930fcafa01379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcf7ef7565040bd8b1930fcafa01379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39421 } } }
I20260926 01:04:47.099113 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dcf7ef7565040bd8b1930fcafa01379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcf7ef7565040bd8b1930fcafa01379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39421 } } }
I20260926 01:04:47.099185 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:47.099207 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:47.099370 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:04:47.099953 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:04:47.101485 1566 catalog_manager.cc:1347] Generated new cluster ID: 4646427c99cb4137b43cf7377abb8160
I20260926 01:04:47.101548 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:47.107944 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.095457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.0.98.62:39421)
0926 01:04:47.095639 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.095648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.095683 (+ 35us) server_negotiation.cc:407] Connection header received
0926 01:04:47.096689 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.096700 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.096768 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.096929 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:47.097644 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.098989 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.100199 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.100777 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.100878 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.103869 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:47.103874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:47.103875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:47.103893 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:47.106322 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.106754 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.106758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.106759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.106884 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.107073 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.107081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.107082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.107340 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:47.107494 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.107591 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.107735 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260926 01:04:47.127506 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:04:47.128087 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:04:47.132246 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379: Generated new TSK 0
I20260926 01:04:47.132440 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:04:47.162292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39421
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:47.245802 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:47.246003 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:47.246037 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:47.247293 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:47.247339 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:47.247367 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:04:47.247393 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:04:47.248988 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:47.249240 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:47.249476 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:47.254372 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:47.254372 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:47.254405 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:47.254630 1579 server_base.cc:1034] running on GCE node
I20260926 01:04:47.254873 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:47.255106 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:47.256249 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790384687256236 us; error 32 us; skew 500 ppm
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:47.258860 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:04:47.259310 1579 webserver.cc:460] Webserver started at http://127.0.98.1:35215/ using document root <none> and password file <none>
I20260926 01:04:47.259524 1579 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:47.259569 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:47.259685 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:47.260733 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "35f0f315077c4415a8ae85cb40f855b8"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "6f1ffc0e2135c3439dcc6aa232a396e8"
server_key_iv: "cf8212bbcc84da0fc5c86e3859f584a5"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.261058 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "35f0f315077c4415a8ae85cb40f855b8"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "6f1ffc0e2135c3439dcc6aa232a396e8"
server_key_iv: "cf8212bbcc84da0fc5c86e3859f584a5"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.262347 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:04:47.263200 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.263373 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:47.263470 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "35f0f315077c4415a8ae85cb40f855b8"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "6f1ffc0e2135c3439dcc6aa232a396e8"
server_key_iv: "cf8212bbcc84da0fc5c86e3859f584a5"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.263566 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:47.274464 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:47.276710 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:47.276831 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:47.277446 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:47.277501 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.277524 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:47.277536 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:47.285171 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46555
I20260926 01:04:47.285547 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:04:47.285566 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46555 every 8 connection(s)
I20260926 01:04:47.287606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 1579
I20260926 01:04:47.287699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:04:47.287865 392 external_mini_cluster.cc:1428] Setting key 4535d6240b1fe969b7e640881889bcc2
I20260926 01:04:47.288522 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.277930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52621 (local address 127.0.98.62:39421)
0926 01:04:47.278038 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.278040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.278208 (+ 168us) server_negotiation.cc:407] Connection header received
0926 01:04:47.278928 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.278929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.278943 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.279029 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:47.279961 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.280525 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.281965 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.282191 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.282250 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.285213 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:47.285216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:47.285216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:47.285230 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:47.286813 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.287314 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.287315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.287316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.287407 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.287726 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.287728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.287728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.287873 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:47.287998 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.288330 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.288430 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 01:04:47.288709 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.277898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39421 (local address 127.0.98.1:52621)
0926 01:04:47.278065 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.278115 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.278807 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.279124 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.279130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.279495 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:47.279848 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.279856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.280765 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.280766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.281870 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.281873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.282264 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.282265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.282422 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.282961 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:47.282969 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:47.285082 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:47.286923 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.286930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.286932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.287204 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.287508 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.287509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.287509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.287630 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.287951 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:47.287958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:47.288177 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.288299 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.288475 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260926 01:04:47.290216 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39421
I20260926 01:04:47.290328 1710 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:47.290608 1710 heartbeater.cc:511] Master 127.0.98.62:39421 requested a full tablet report, sending...
I20260926 01:04:47.291217 1506 ts_manager.cc:194] Registered new tserver with Master: 35f0f315077c4415a8ae85cb40f855b8 (127.0.98.1:46555)
I20260926 01:04:47.292076 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52621
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:04:47.296126 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.290355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44807 (local address 127.0.98.62:39421)
0926 01:04:47.290422 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.290424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.290721 (+ 297us) server_negotiation.cc:407] Connection header received
0926 01:04:47.290748 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.290749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.290761 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.290809 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:47.291180 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.291674 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.292404 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.292697 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.292734 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.293690 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:47.293692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:47.293692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:47.293700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:47.295017 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.295325 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.295326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.295326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.295382 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.295622 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.295623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.295623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.295736 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:47.295784 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.295947 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.296030 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:04:47.296160 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.290455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39421 (local address 127.0.98.1:44807)
0926 01:04:47.290606 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.290616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.290675 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.290858 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.290859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.291047 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:47.291110 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.291112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.291766 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.291767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.292319 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.292321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.292802 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.292803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.292854 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.293288 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:47.293293 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:47.293605 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:47.295094 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.295096 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.295096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.295264 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.295461 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.295462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.295462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.295539 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.295793 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:47.295794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:47.295849 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.295920 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.296038 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:04:47.314561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39421
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:47.395035 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:47.395246 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:47.395272 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:47.396569 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:47.396620 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:47.396641 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:04:47.396659 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:04:47.398334 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:47.398669 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:47.398898 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:47.403985 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:47.404018 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:47.404029 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:47.404606 1719 server_base.cc:1034] running on GCE node
I20260926 01:04:47.404841 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:47.405153 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:47.406318 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790384687406301 us; error 29 us; skew 500 ppm
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:47.409006 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:04:47.409499 1719 webserver.cc:460] Webserver started at http://127.0.98.2:37069/ using document root <none> and password file <none>
I20260926 01:04:47.409725 1719 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:47.409775 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:47.409902 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:47.411005 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "9e5b233002548a67b59c48cad115bb92"
server_key_iv: "e5cbf103b6b205baa469ae9dec046048"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.411327 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "9e5b233002548a67b59c48cad115bb92"
server_key_iv: "e5cbf103b6b205baa469ae9dec046048"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.412557 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:04:47.413266 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.413434 1719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:04:47.413518 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "9e5b233002548a67b59c48cad115bb92"
server_key_iv: "e5cbf103b6b205baa469ae9dec046048"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.413581 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:47.431721 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:47.433614 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:47.433760 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:47.434314 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:47.434355 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.434409 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:47.434450 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:47.441578 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38645
I20260926 01:04:47.441965 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:04:47.442399 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38645 every 8 connection(s)
I20260926 01:04:47.443853 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.434759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38225 (local address 127.0.98.62:39421)
0926 01:04:47.434825 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.434827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.435056 (+ 229us) server_negotiation.cc:407] Connection header received
0926 01:04:47.435748 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.435749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.435763 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.435834 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:47.436703 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.437207 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.438350 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.438591 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.438652 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.441151 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:47.441153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:47.441153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:47.441162 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:47.442715 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.443125 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.443126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.443127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.443180 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.443377 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.443377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.443377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.443477 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:47.443522 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.443730 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.443790 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:04:47.444061 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.434739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39421 (local address 127.0.98.2:38225)
0926 01:04:47.434896 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.434945 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.435623 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.435916 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.435921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.436304 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:47.436615 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.436623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.437311 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.437312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.438254 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.438257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.438690 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.438692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.438828 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.439418 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:47.439430 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:47.441046 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:47.442785 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.442788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.442790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.443052 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.443230 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.443230 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.443231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.443326 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.443519 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:47.443522 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:47.443643 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.443718 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.443838 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260926 01:04:47.444531 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39421
I20260926 01:04:47.444613 1850 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:47.444799 1850 heartbeater.cc:511] Master 127.0.98.62:39421 requested a full tablet report, sending...
I20260926 01:04:47.445196 1506 ts_manager.cc:194] Registered new tserver with Master: 33d94adc8a5f4dc1820bdf4c94d0f77f (127.0.98.2:38645)
I20260926 01:04:47.445662 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38225
I20260926 01:04:47.449493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 1719
I20260926 01:04:47.449579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:04:47.449738 392 external_mini_cluster.cc:1428] Setting key b471091a287ea04d9fb662e0fb3f91b8
I20260926 01:04:47.450912 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.445536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39421 (local address 127.0.98.2:39217)
0926 01:04:47.445663 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.445674 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.445735 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.445841 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.445842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.446027 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:47.446108 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.446111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.446773 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.446774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.447285 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.447286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.447621 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.447623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.447688 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.448077 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:47.448081 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:47.448357 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:47.449725 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.449726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.449727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.449933 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.450145 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.450146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.450146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.450210 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.450614 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:47.450615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:47.450662 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.450718 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.450809 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260926 01:04:47.450917 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.445602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39217 (local address 127.0.98.62:39421)
0926 01:04:47.445647 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.445649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.445746 (+ 97us) server_negotiation.cc:407] Connection header received
0926 01:04:47.445783 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.445783 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.445793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.445837 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:47.446180 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.446694 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.447358 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.447537 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.447589 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.448428 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:47.448429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:47.448429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:47.448435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:47.449654 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.450006 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.450007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.450008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.450077 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.450264 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.450265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.450265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.450498 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:47.450613 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.450754 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.450832 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:04:47.478056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39421
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:47.559331 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:47.559557 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:47.559582 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:47.560772 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:47.560817 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:47.560837 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:04:47.560853 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:04:47.562479 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:47.562693 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:47.562886 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:47.568079 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:47.568081 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:47.568079 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:47.568267 1859 server_base.cc:1034] running on GCE node
I20260926 01:04:47.568550 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:47.568799 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:47.569960 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790384687569946 us; error 37 us; skew 500 ppm
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:47.572593 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:04:47.573042 1859 webserver.cc:460] Webserver started at http://127.0.98.3:45955/ using document root <none> and password file <none>
I20260926 01:04:47.573268 1859 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:47.573316 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:47.573441 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:47.574607 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c254457ba5a4f3e8907589375637145"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "397ebb14873a88486d609865aa534b4c"
server_key_iv: "dcc5fe98cbda096fbff7789c8da96a50"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.574939 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c254457ba5a4f3e8907589375637145"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "397ebb14873a88486d609865aa534b4c"
server_key_iv: "dcc5fe98cbda096fbff7789c8da96a50"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.576220 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:47.576892 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.577037 1859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:04:47.577116 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "9c254457ba5a4f3e8907589375637145"
format_stamp: "Formatted at 2026-09-26 01:04:47 on dist-test-slave-l73v"
server_key: "397ebb14873a88486d609865aa534b4c"
server_key_iv: "dcc5fe98cbda096fbff7789c8da96a50"
server_key_version: "encryptionkey@0"
I20260926 01:04:47.577181 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:47.598438 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:47.600862 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:47.600993 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:47.601507 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:47.601549 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.601573 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:47.601593 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:47.608661 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40353
I20260926 01:04:47.608734 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40353 every 8 connection(s)
I20260926 01:04:47.609056 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:04:47.611569 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.601942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60437 (local address 127.0.98.62:39421)
0926 01:04:47.601981 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.601983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.602248 (+ 265us) server_negotiation.cc:407] Connection header received
0926 01:04:47.602792 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.602793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.602805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.602857 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:47.603674 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.604429 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.606016 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.606315 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.606418 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.609050 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:47.609052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:47.609052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:47.609061 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:47.610369 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.610758 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.610759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.610759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.610818 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.611054 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.611055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.611055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.611169 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:47.611227 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.611433 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.611506 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:04:47.611712 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.601955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39421 (local address 127.0.98.3:60437)
0926 01:04:47.602141 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.602175 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.602700 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.602993 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.602997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.603308 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:47.603584 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.603591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.604535 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.604536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.605911 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.605916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.606443 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.606444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.606599 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.607186 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:47.607197 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:47.608950 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:47.610474 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.610479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.610481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.610693 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.610876 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.610876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.610877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.610966 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.611219 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:47.611222 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:47.611334 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.611409 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.611526 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260926 01:04:47.612185 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39421
I20260926 01:04:47.612264 1990 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:47.612443 1990 heartbeater.cc:511] Master 127.0.98.62:39421 requested a full tablet report, sending...
I20260926 01:04:47.612829 1507 ts_manager.cc:194] Registered new tserver with Master: 9c254457ba5a4f3e8907589375637145 (127.0.98.3:40353)
I20260926 01:04:47.613095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 1859
I20260926 01:04:47.613186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:04:47.613354 392 external_mini_cluster.cc:1428] Setting key 1354913ead10a262474ab24f80796166
I20260926 01:04:47.613369 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60437
I20260926 01:04:47.614272 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:04:47.618928 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.613266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39421 (local address 127.0.98.3:44867)
0926 01:04:47.613432 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.613444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.613496 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.613673 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.613674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.613861 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:47.613940 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.613942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.614764 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.614765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.615473 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.615475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.615772 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.615773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.615830 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.616323 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:47.616328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:47.616600 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:47.618023 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.618024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.618025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.618186 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.618350 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.618350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.618351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.618438 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.618661 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:47.618662 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:47.618705 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.618766 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.618853 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":5,"thread_start_us":93,"threads_started":1}
I20260926 01:04:47.618961 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.613269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44867 (local address 127.0.98.62:39421)
0926 01:04:47.613485 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.613486 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.613495 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:04:47.613569 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.613570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.613580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.613606 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:47.614007 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.614677 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.615546 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.615718 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.615758 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.616673 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:47.616674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:47.616674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:47.616682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:47.617953 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.618245 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.618246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.618246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.618299 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.618483 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.618484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.618484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.618605 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:47.618668 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.618802 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.618875 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:47.645368 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.637834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.0.98.62:39421)
0926 01:04:47.637920 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.637922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.637935 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:04:47.637967 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.637969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.637982 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.638075 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:47.638813 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.639540 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.640346 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.640523 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.640564 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.642945 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:47.642947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:47.642947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:47.642955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:47.644172 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.644627 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.644628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.644628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.644689 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.644857 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.644857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.644858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.644959 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:47.645057 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.645228 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.645299 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:04:47.647128 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:04:47.648722 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:04:47.655316 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.651304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46555 (local address 127.0.0.1:42456)
0926 01:04:47.651687 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.651696 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.651754 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.651966 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.651967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.651969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:47.652016 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.652018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.653247 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.653248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.654339 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.654340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.655002 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.655003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.655082 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.655097 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.655163 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.655218 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":24,"thread_start_us":55,"threads_started":1}
I20260926 01:04:47.655395 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.651313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.0.98.3:40353)
0926 01:04:47.651862 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.651866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.651875 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:04:47.651908 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.651910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.651925 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.652137 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:47.652196 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.652874 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.654346 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.655071 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.655130 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.655198 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.655256 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.655295 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":80,"threads_started":1}
I20260926 01:04:47.655560 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.651247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.0.98.2:38645)
0926 01:04:47.651571 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.651575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.651586 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:04:47.651625 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.651627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.651643 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.651728 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:47.651973 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.653120 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.654275 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.655017 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.655086 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.655177 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.655424 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.655463 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":71,"threads_started":1}
I20260926 01:04:47.655680 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.651500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.0.98.1:46555)
0926 01:04:47.651670 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.651674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.651736 (+ 62us) server_negotiation.cc:407] Connection header received
0926 01:04:47.651813 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.651815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.651830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.651902 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:47.652086 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.653172 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.654423 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.654924 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.654990 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.655075 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.655534 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.655576 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260926 01:04:47.655380 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.651115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40353 (local address 127.0.0.1:54624)
0926 01:04:47.651406 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.651416 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.651465 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.652014 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.652015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.652017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:47.652066 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.652067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.652956 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.652957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.654281 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.654283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.655131 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.655132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.655195 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.655198 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.655244 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.655276 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":91,"threads_started":1}
I20260926 01:04:47.655920 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.651108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38645 (local address 127.0.0.1:47254)
0926 01:04:47.651264 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.651307 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.651374 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.651737 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.651741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.651743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:47.651817 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.651819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.653228 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.653229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.654201 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.654204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.655215 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.655216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.655286 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.655290 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.655340 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.655381 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":35,"thread_start_us":51,"threads_started":1}
I20260926 01:04:47.656412 1644 tablet_service.cc:1467] Processing CreateTablet for tablet d41b7bae82e74ab294fd4f2372e425bc (DEFAULT_TABLE table=test-table [id=4f8d50fa682d432785282cf5eadafae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:47.656412 1924 tablet_service.cc:1467] Processing CreateTablet for tablet d41b7bae82e74ab294fd4f2372e425bc (DEFAULT_TABLE table=test-table [id=4f8d50fa682d432785282cf5eadafae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:47.656777 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d41b7bae82e74ab294fd4f2372e425bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:47.656777 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d41b7bae82e74ab294fd4f2372e425bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:47.659070 1784 tablet_service.cc:1467] Processing CreateTablet for tablet d41b7bae82e74ab294fd4f2372e425bc (DEFAULT_TABLE table=test-table [id=4f8d50fa682d432785282cf5eadafae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:47.659403 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d41b7bae82e74ab294fd4f2372e425bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:47.660495 2013 tablet_bootstrap.cc:492] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: Bootstrap starting.
I20260926 01:04:47.660813 2014 tablet_bootstrap.cc:492] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: Bootstrap starting.
I20260926 01:04:47.661222 2013 tablet_bootstrap.cc:654] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:47.661324 2014 tablet_bootstrap.cc:654] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:47.661553 2013 log.cc:824] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:47.661599 2014 log.cc:824] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:47.662220 2014 tablet_bootstrap.cc:492] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: No bootstrap required, opened a new log
I20260926 01:04:47.662277 2014 ts_tablet_manager.cc:1397] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:04:47.662961 2013 tablet_bootstrap.cc:492] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: No bootstrap required, opened a new log
I20260926 01:04:47.663017 2013 ts_tablet_manager.cc:1397] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 01:04:47.663797 2014 raft_consensus.cc:348] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } }
I20260926 01:04:47.663935 2014 raft_consensus.cc:374] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:47.663964 2014 raft_consensus.cc:729] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f0f315077c4415a8ae85cb40f855b8, State: Initialized, Role: FOLLOWER
I20260926 01:04:47.664089 2014 consensus_queue.cc:260] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } }
I20260926 01:04:47.664353 2014 ts_tablet_manager.cc:1428] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:04:47.664405 1710 heartbeater.cc:503] Master 127.0.98.62:39421 was elected leader, sending a full tablet report...
I20260926 01:04:47.664549 2017 tablet_bootstrap.cc:492] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: Bootstrap starting.
I20260926 01:04:47.664515 2013 raft_consensus.cc:348] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } }
I20260926 01:04:47.664667 2013 raft_consensus.cc:374] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:47.664709 2013 raft_consensus.cc:729] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c254457ba5a4f3e8907589375637145, State: Initialized, Role: FOLLOWER
I20260926 01:04:47.664830 2013 consensus_queue.cc:260] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } }
I20260926 01:04:47.665104 2013 ts_tablet_manager.cc:1428] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:04:47.665208 1990 heartbeater.cc:503] Master 127.0.98.62:39421 was elected leader, sending a full tablet report...
I20260926 01:04:47.665237 2017 tablet_bootstrap.cc:654] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:47.665562 2017 log.cc:824] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:47.666304 2017 tablet_bootstrap.cc:492] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: No bootstrap required, opened a new log
I20260926 01:04:47.666361 2017 ts_tablet_manager.cc:1397] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:04:47.667889 2017 raft_consensus.cc:348] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } }
I20260926 01:04:47.668021 2017 raft_consensus.cc:374] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:47.668042 2017 raft_consensus.cc:729] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d94adc8a5f4dc1820bdf4c94d0f77f, State: Initialized, Role: FOLLOWER
I20260926 01:04:47.668128 2017 consensus_queue.cc:260] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } }
I20260926 01:04:47.668363 2017 ts_tablet_manager.cc:1428] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:04:47.668433 1850 heartbeater.cc:503] Master 127.0.98.62:39421 was elected leader, sending a full tablet report...
W20260926 01:04:47.692533 1851 tablet.cc:2369] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:04:47.786336 1711 tablet.cc:2369] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:04:47.787459 2018 raft_consensus.cc:482] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:04:47.787580 2018 raft_consensus.cc:504] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } }
I20260926 01:04:47.787938 2018 leader_election.cc:288] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33d94adc8a5f4dc1820bdf4c94d0f77f (127.0.98.2:38645), 9c254457ba5a4f3e8907589375637145 (127.0.98.3:40353)
I20260926 01:04:47.791090 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.788144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38645 (local address 127.0.98.1:35919)
0926 01:04:47.788229 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.788241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.788290 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.788452 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.788453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.788455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:47.788519 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.788520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.789131 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.789132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.790053 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.790056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.790872 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.790873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.790937 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.790940 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.790986 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.791021 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":21}
I20260926 01:04:47.791087 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.788208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35919 (local address 127.0.98.2:38645)
0926 01:04:47.788258 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.788260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.788274 (+ 14us) server_negotiation.cc:407] Connection header received
0926 01:04:47.788373 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.788374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.788386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.788435 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:47.788601 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.789066 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.790136 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.790775 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.790847 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.790858 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.791001 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.791033 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:04:47.791431 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.788195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40353 (local address 127.0.98.1:55983)
0926 01:04:47.788331 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.788341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.788403 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.788533 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.788534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.788535 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:47.788604 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.788606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.789310 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.789311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.790139 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.790142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.791200 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.791201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.791267 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.791270 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.791318 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.791352 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":14,"thread_start_us":66,"threads_started":1}
I20260926 01:04:47.791427 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.788277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55983 (local address 127.0.98.3:40353)
0926 01:04:47.788321 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.788323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.788407 (+ 84us) server_negotiation.cc:407] Connection header received
0926 01:04:47.788454 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.788455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.788465 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.788524 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:47.788666 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.789233 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.790205 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.790775 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.790847 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.790858 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.791324 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.791368 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:04:47.791497 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41b7bae82e74ab294fd4f2372e425bc" candidate_uuid: "35f0f315077c4415a8ae85cb40f855b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" is_pre_election: true
I20260926 01:04:47.791626 1804 raft_consensus.cc:2454] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f0f315077c4415a8ae85cb40f855b8 in term 0.
I20260926 01:04:47.791844 1595 leader_election.cc:302] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d94adc8a5f4dc1820bdf4c94d0f77f, 35f0f315077c4415a8ae85cb40f855b8; no voters:
I20260926 01:04:47.791983 2018 raft_consensus.cc:2781] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:04:47.792038 2018 raft_consensus.cc:482] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:04:47.791973 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41b7bae82e74ab294fd4f2372e425bc" candidate_uuid: "35f0f315077c4415a8ae85cb40f855b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c254457ba5a4f3e8907589375637145" is_pre_election: true
I20260926 01:04:47.792061 2018 raft_consensus.cc:3037] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:47.792124 1944 raft_consensus.cc:2454] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f0f315077c4415a8ae85cb40f855b8 in term 0.
I20260926 01:04:47.793030 2018 raft_consensus.cc:504] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } }
I20260926 01:04:47.793176 2018 leader_election.cc:288] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [CANDIDATE]: Term 1 election: Requested vote from peers 33d94adc8a5f4dc1820bdf4c94d0f77f (127.0.98.2:38645), 9c254457ba5a4f3e8907589375637145 (127.0.98.3:40353)
I20260926 01:04:47.793356 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41b7bae82e74ab294fd4f2372e425bc" candidate_uuid: "35f0f315077c4415a8ae85cb40f855b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f"
I20260926 01:04:47.793356 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41b7bae82e74ab294fd4f2372e425bc" candidate_uuid: "35f0f315077c4415a8ae85cb40f855b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c254457ba5a4f3e8907589375637145"
I20260926 01:04:47.793448 1944 raft_consensus.cc:3037] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:47.793448 1804 raft_consensus.cc:3037] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:47.794785 1804 raft_consensus.cc:2454] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f0f315077c4415a8ae85cb40f855b8 in term 1.
I20260926 01:04:47.794854 1944 raft_consensus.cc:2454] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f0f315077c4415a8ae85cb40f855b8 in term 1.
I20260926 01:04:47.795024 1595 leader_election.cc:302] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d94adc8a5f4dc1820bdf4c94d0f77f, 35f0f315077c4415a8ae85cb40f855b8; no voters:
I20260926 01:04:47.795137 2018 raft_consensus.cc:2781] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:47.795295 2018 raft_consensus.cc:686] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 1 LEADER]: Becoming Leader. State: Replica: 35f0f315077c4415a8ae85cb40f855b8, State: Running, Role: LEADER
I20260926 01:04:47.795385 2018 consensus_queue.cc:237] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } }
I20260926 01:04:47.796288 1506 catalog_manager.cc:5495] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f0f315077c4415a8ae85cb40f855b8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "35f0f315077c4415a8ae85cb40f855b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } health_report { overall_health: HEALTHY } } }
I20260926 01:04:47.805318 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.802771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.0.98.1:46555)
0926 01:04:47.802844 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.802846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.802855 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:04:47.802890 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.802891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.802911 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.802975 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:47.803138 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.803616 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.804445 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.804649 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.804721 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.804925 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:47.805037 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:47.805113 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.805233 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.805254 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 01:04:47.807188 1944 raft_consensus.cc:1261] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145 [term 1 FOLLOWER]: Refusing update from remote peer 35f0f315077c4415a8ae85cb40f855b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:47.807188 1804 raft_consensus.cc:1261] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 1 FOLLOWER]: Refusing update from remote peer 35f0f315077c4415a8ae85cb40f855b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:47.807570 2023 consensus_queue.cc:1035] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:47.807725 2018 consensus_queue.cc:1035] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:47.810156 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333415681255272448 to 7333415681210515456. Current Snapshot: MvccSnapshot[applied={T|T < 7333415681255272448}]
I20260926 01:04:47.810648 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333415681255272448 to 7333415681210515456. Current Snapshot: MvccSnapshot[applied={T|T < 7333415681255272448}]
I20260926 01:04:47.813831 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:47.813952 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:47.814811 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379: Sending DeleteTablet for 3 replicas of tablet d41b7bae82e74ab294fd4f2372e425bc
I20260926 01:04:47.815150 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet d41b7bae82e74ab294fd4f2372e425bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47254
I20260926 01:04:47.815225 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet d41b7bae82e74ab294fd4f2372e425bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42456
I20260926 01:04:47.815241 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet d41b7bae82e74ab294fd4f2372e425bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54624
I20260926 01:04:47.815723 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:04:47.815862 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:04:47.815951 2036 tablet_replica.cc:330] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: stopping tablet replica
I20260926 01:04:47.816111 2036 raft_consensus.cc:2229] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:47.816367 2036 raft_consensus.cc:2258] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:47.816632 2037 tablet_replica.cc:330] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: stopping tablet replica
I20260926 01:04:47.816778 2038 tablet_replica.cc:330] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: stopping tablet replica
I20260926 01:04:47.816917 2037 raft_consensus.cc:2229] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:04:47.817059 2038 raft_consensus.cc:2229] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:47.817176 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 09e74c4f47e843129aee53c0e14f8cc8 (DEFAULT_TABLE table=test-table [id=550825db63b94473b9a1dd474f00e4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:47.817322 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e74c4f47e843129aee53c0e14f8cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:47.817428 2038 raft_consensus.cc:2258] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:47.817552 2037 raft_consensus.cc:2258] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:47.817493 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 09e74c4f47e843129aee53c0e14f8cc8 (DEFAULT_TABLE table=test-table [id=550825db63b94473b9a1dd474f00e4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:47.817721 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e74c4f47e843129aee53c0e14f8cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:47.818533 2017 tablet_bootstrap.cc:492] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f: Bootstrap starting.
I20260926 01:04:47.818879 2017 tablet_bootstrap.cc:654] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:47.819093 2013 tablet_bootstrap.cc:492] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: Bootstrap starting.
I20260926 01:04:47.817620 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 09e74c4f47e843129aee53c0e14f8cc8 (DEFAULT_TABLE table=test-table [id=550825db63b94473b9a1dd474f00e4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:47.819473 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e74c4f47e843129aee53c0e14f8cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:47.819514 2013 tablet_bootstrap.cc:654] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:47.819626 2036 ts_tablet_manager.cc:1905] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:47.820797 2014 tablet_bootstrap.cc:492] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8: Bootstrap starting.
I20260926 01:04:47.821183 2014 tablet_bootstrap.cc:654] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:47.821285 2038 ts_tablet_manager.cc:1905] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:47.821812 2017 tablet_bootstrap.cc:492] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f: No bootstrap required, opened a new log
I20260926 01:04:47.821863 2017 ts_tablet_manager.cc:1397] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 01:04:47.821965 2014 tablet_bootstrap.cc:492] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8: No bootstrap required, opened a new log
I20260926 01:04:47.822000 2014 ts_tablet_manager.cc:1397] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:47.822050 2017 raft_consensus.cc:348] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.822137 2017 raft_consensus.cc:374] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:47.822156 2017 raft_consensus.cc:729] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d94adc8a5f4dc1820bdf4c94d0f77f, State: Initialized, Role: FOLLOWER
I20260926 01:04:47.822127 2014 raft_consensus.cc:348] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.822185 2014 raft_consensus.cc:374] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:47.822202 2014 raft_consensus.cc:729] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f0f315077c4415a8ae85cb40f855b8, State: Initialized, Role: FOLLOWER
I20260926 01:04:47.822201 2017 consensus_queue.cc:260] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.822254 2014 consensus_queue.cc:260] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.822283 2017 ts_tablet_manager.cc:1428] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.822335 2014 ts_tablet_manager.cc:1428] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.822841 2038 ts_tablet_manager.cc:1918] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:04:47.822968 2036 ts_tablet_manager.cc:1918] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:04:47.823015 2038 log.cc:1196] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/d41b7bae82e74ab294fd4f2372e425bc
I20260926 01:04:47.823041 2036 log.cc:1196] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/d41b7bae82e74ab294fd4f2372e425bc
I20260926 01:04:47.823294 2036 ts_tablet_manager.cc:1939] T d41b7bae82e74ab294fd4f2372e425bc P 33d94adc8a5f4dc1820bdf4c94d0f77f: Deleting consensus metadata
I20260926 01:04:47.823292 2038 ts_tablet_manager.cc:1939] T d41b7bae82e74ab294fd4f2372e425bc P 9c254457ba5a4f3e8907589375637145: Deleting consensus metadata
I20260926 01:04:47.823693 1491 catalog_manager.cc:4841] TS 33d94adc8a5f4dc1820bdf4c94d0f77f (127.0.98.2:38645): tablet d41b7bae82e74ab294fd4f2372e425bc (table test-table [id=4f8d50fa682d432785282cf5eadafae2]) successfully deleted
I20260926 01:04:47.823695 1493 catalog_manager.cc:4841] TS 9c254457ba5a4f3e8907589375637145 (127.0.98.3:40353): tablet d41b7bae82e74ab294fd4f2372e425bc (table test-table [id=4f8d50fa682d432785282cf5eadafae2]) successfully deleted
I20260926 01:04:47.823938 2037 ts_tablet_manager.cc:1905] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:47.825479 2037 ts_tablet_manager.cc:1918] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:04:47.825563 2037 log.cc:1196] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/d41b7bae82e74ab294fd4f2372e425bc
I20260926 01:04:47.825800 2037 ts_tablet_manager.cc:1939] T d41b7bae82e74ab294fd4f2372e425bc P 35f0f315077c4415a8ae85cb40f855b8: Deleting consensus metadata
I20260926 01:04:47.826613 1493 catalog_manager.cc:4841] TS 35f0f315077c4415a8ae85cb40f855b8 (127.0.98.1:46555): tablet d41b7bae82e74ab294fd4f2372e425bc (table test-table [id=4f8d50fa682d432785282cf5eadafae2]) successfully deleted
I20260926 01:04:47.826799 2013 tablet_bootstrap.cc:492] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: No bootstrap required, opened a new log
I20260926 01:04:47.826853 2013 ts_tablet_manager.cc:1397] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260926 01:04:47.826997 2013 raft_consensus.cc:348] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.827064 2013 raft_consensus.cc:374] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:47.827076 2013 raft_consensus.cc:729] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c254457ba5a4f3e8907589375637145, State: Initialized, Role: FOLLOWER
I20260926 01:04:47.827116 2013 consensus_queue.cc:260] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.827208 2013 ts_tablet_manager.cc:1428] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.828467 2019 raft_consensus.cc:482] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:04:47.828538 2019 raft_consensus.cc:504] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.828862 2019 leader_election.cc:288] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f0f315077c4415a8ae85cb40f855b8 (127.0.98.1:46555), 33d94adc8a5f4dc1820bdf4c94d0f77f (127.0.98.2:38645)
I20260926 01:04:47.832540 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.829091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60569 (local address 127.0.98.1:46555)
0926 01:04:47.829134 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.829137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.829275 (+ 138us) server_negotiation.cc:407] Connection header received
0926 01:04:47.829489 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.829490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.829517 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.829559 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:47.829764 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.830217 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.831735 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.832163 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.832228 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.832241 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.832418 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.832462 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:04:47.832608 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.829052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38645 (local address 127.0.98.3:60627)
0926 01:04:47.829153 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.829164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.829203 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.829438 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.829439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.829441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:47.829509 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.829510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.830466 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.830467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.831612 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.831615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.832373 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.832374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.832447 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.832450 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.832498 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.832543 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":3}
I20260926 01:04:47.832623 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.829291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60627 (local address 127.0.98.2:38645)
0926 01:04:47.829341 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.829343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.829350 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:04:47.829375 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.829376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.829384 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.829434 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:47.829587 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.830354 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.831711 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.832295 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.832369 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.832379 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.832512 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.832549 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:04:47.833048 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e74c4f47e843129aee53c0e14f8cc8" candidate_uuid: "9c254457ba5a4f3e8907589375637145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" is_pre_election: true
I20260926 01:04:47.833140 1804 raft_consensus.cc:2454] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c254457ba5a4f3e8907589375637145 in term 0.
I20260926 01:04:47.833367 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.829053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46555 (local address 127.0.98.3:60569)
0926 01:04:47.829217 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.829226 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.829285 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.829601 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.829602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.829604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:47.829688 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.829690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.830299 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.830300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.831611 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.831615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.832256 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.832258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.832334 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.832337 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.833207 (+ 870us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.833268 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":90,"thread_start_us":64,"threads_started":1}
I20260926 01:04:47.833334 1875 leader_election.cc:302] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d94adc8a5f4dc1820bdf4c94d0f77f, 9c254457ba5a4f3e8907589375637145; no voters:
I20260926 01:04:47.833578 2019 raft_consensus.cc:2781] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:04:47.833636 2019 raft_consensus.cc:482] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:04:47.833664 2019 raft_consensus.cc:3037] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:47.834343 2019 raft_consensus.cc:504] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.834512 2019 leader_election.cc:288] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [CANDIDATE]: Term 1 election: Requested vote from peers 35f0f315077c4415a8ae85cb40f855b8 (127.0.98.1:46555), 33d94adc8a5f4dc1820bdf4c94d0f77f (127.0.98.2:38645)
I20260926 01:04:47.834513 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e74c4f47e843129aee53c0e14f8cc8" candidate_uuid: "9c254457ba5a4f3e8907589375637145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f0f315077c4415a8ae85cb40f855b8" is_pre_election: true
I20260926 01:04:47.834666 1664 raft_consensus.cc:2454] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c254457ba5a4f3e8907589375637145 in term 0.
I20260926 01:04:47.834759 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e74c4f47e843129aee53c0e14f8cc8" candidate_uuid: "9c254457ba5a4f3e8907589375637145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f0f315077c4415a8ae85cb40f855b8"
I20260926 01:04:47.834837 1663 raft_consensus.cc:3037] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:47.835529 1663 raft_consensus.cc:2454] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c254457ba5a4f3e8907589375637145 in term 1.
I20260926 01:04:47.835613 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e74c4f47e843129aee53c0e14f8cc8" candidate_uuid: "9c254457ba5a4f3e8907589375637145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f"
I20260926 01:04:47.835672 1804 raft_consensus.cc:3037] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:47.836187 1804 raft_consensus.cc:2454] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c254457ba5a4f3e8907589375637145 in term 1.
I20260926 01:04:47.836366 1875 leader_election.cc:302] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d94adc8a5f4dc1820bdf4c94d0f77f, 9c254457ba5a4f3e8907589375637145; no voters:
I20260926 01:04:47.836472 2019 raft_consensus.cc:2781] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:47.836632 2019 raft_consensus.cc:686] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 1 LEADER]: Becoming Leader. State: Replica: 9c254457ba5a4f3e8907589375637145, State: Running, Role: LEADER
I20260926 01:04:47.836710 2019 consensus_queue.cc:237] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.837401 1506 catalog_manager.cc:5495] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c254457ba5a4f3e8907589375637145 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c254457ba5a4f3e8907589375637145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } health_report { overall_health: HEALTHY } } }
W20260926 01:04:47.859946 1991 tablet.cc:2369] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 01:04:47 dist-test-slave-l73v krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:47.872056 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.863164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39421 (local address 127.0.0.1:45138)
0926 01:04:47.863361 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:47.863374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:47.863438 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:47.863633 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:47.863634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:47.863950 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:47.863999 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.864001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.865076 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.865077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.865712 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:47.865714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.866464 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.866466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:47.866535 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.867106 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:47.867111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:47.869022 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:47.870939 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.870950 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.870953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.871202 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.871412 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:47.871412 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:47.871412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:47.871472 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:47.871753 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:47.871762 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:47.871815 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 01:04:47.871874 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:04:47.871978 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":126,"threads_started":1}
I20260926 01:04:47.872167 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:47.863270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.0.98.62:39421)
0926 01:04:47.863343 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:47.863346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:47.863442 (+ 96us) server_negotiation.cc:407] Connection header received
0926 01:04:47.863478 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:47.863479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:47.863490 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:47.863575 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:47.864123 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.864958 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.865789 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:47.866085 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:47.866476 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:47.869128 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:47.869130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:47.869130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:47.869146 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:47.870842 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.871282 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.871284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.871284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.871349 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:47.871527 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:47.871528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:47.871528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:47.871684 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:47.871752 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:47.871912 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 01:04:47.872044 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":45}
I20260926 01:04:47.874457 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45138:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 01:04:47.874718 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:04:47.876077 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 71592338ba544d248d9df5251f950f9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cce4f45b6924ea2a34cf18af0e27428]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:04:47.876127 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 71592338ba544d248d9df5251f950f9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cce4f45b6924ea2a34cf18af0e27428]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:04:47.876237 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71592338ba544d248d9df5251f950f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:47.876284 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71592338ba544d248d9df5251f950f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:47.877564 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 71592338ba544d248d9df5251f950f9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cce4f45b6924ea2a34cf18af0e27428]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:04:47.877672 2014 tablet_bootstrap.cc:492] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: Bootstrap starting.
I20260926 01:04:47.877707 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71592338ba544d248d9df5251f950f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:47.878048 2014 tablet_bootstrap.cc:654] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:47.878743 2014 tablet_bootstrap.cc:492] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: No bootstrap required, opened a new log
I20260926 01:04:47.878800 2014 ts_tablet_manager.cc:1397] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:04:47.878957 2014 raft_consensus.cc:348] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.879052 2014 raft_consensus.cc:374] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:47.879073 2014 raft_consensus.cc:729] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f0f315077c4415a8ae85cb40f855b8, State: Initialized, Role: FOLLOWER
I20260926 01:04:47.879119 2014 consensus_queue.cc:260] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.879225 2018 tablet_replica.cc:439] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } }
I20260926 01:04:47.879331 2018 tablet_replica.cc:442] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:47.879328 2017 tablet_bootstrap.cc:492] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: Bootstrap starting.
I20260926 01:04:47.879227 2014 ts_tablet_manager.cc:1428] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 01:04:47.879760 2017 tablet_bootstrap.cc:654] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:47.880607 2013 tablet_bootstrap.cc:492] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: Bootstrap starting.
I20260926 01:04:47.880936 2013 tablet_bootstrap.cc:654] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:47.881584 2013 tablet_bootstrap.cc:492] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: No bootstrap required, opened a new log
I20260926 01:04:47.881630 2013 ts_tablet_manager.cc:1397] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:47.881975 2013 raft_consensus.cc:348] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.882046 2013 raft_consensus.cc:374] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:47.882066 2013 raft_consensus.cc:729] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c254457ba5a4f3e8907589375637145, State: Initialized, Role: FOLLOWER
I20260926 01:04:47.882107 2013 consensus_queue.cc:260] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.882195 2013 ts_tablet_manager.cc:1428] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:47.882313 2019 tablet_replica.cc:439] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } }
I20260926 01:04:47.882364 2019 tablet_replica.cc:442] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:47.882844 2017 tablet_bootstrap.cc:492] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: No bootstrap required, opened a new log
I20260926 01:04:47.882906 2017 ts_tablet_manager.cc:1397] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 01:04:47.883068 2017 raft_consensus.cc:348] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.883149 2017 raft_consensus.cc:374] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:47.883174 2017 raft_consensus.cc:729] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d94adc8a5f4dc1820bdf4c94d0f77f, State: Initialized, Role: FOLLOWER
I20260926 01:04:47.883220 2017 consensus_queue.cc:260] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:47.883306 2017 ts_tablet_manager.cc:1428] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:47.883386 2021 tablet_replica.cc:439] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } }
I20260926 01:04:47.883495 2021 tablet_replica.cc:442] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:48.165323 2019 raft_consensus.cc:482] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:04:48.165423 2019 raft_consensus.cc:504] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:48.165593 2019 leader_election.cc:288] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f0f315077c4415a8ae85cb40f855b8 (127.0.98.1:46555), 33d94adc8a5f4dc1820bdf4c94d0f77f (127.0.98.2:38645)
I20260926 01:04:48.165773 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71592338ba544d248d9df5251f950f9d" candidate_uuid: "9c254457ba5a4f3e8907589375637145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" is_pre_election: true
I20260926 01:04:48.165891 1804 raft_consensus.cc:2454] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c254457ba5a4f3e8907589375637145 in term 0.
I20260926 01:04:48.165820 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71592338ba544d248d9df5251f950f9d" candidate_uuid: "9c254457ba5a4f3e8907589375637145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f0f315077c4415a8ae85cb40f855b8" is_pre_election: true
I20260926 01:04:48.165941 1663 raft_consensus.cc:2454] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c254457ba5a4f3e8907589375637145 in term 0.
I20260926 01:04:48.166038 1875 leader_election.cc:302] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d94adc8a5f4dc1820bdf4c94d0f77f, 9c254457ba5a4f3e8907589375637145; no voters:
I20260926 01:04:48.166162 2019 raft_consensus.cc:2781] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:04:48.166209 2019 raft_consensus.cc:482] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:04:48.166236 2019 raft_consensus.cc:3037] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:48.166994 2019 raft_consensus.cc:504] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:48.167156 2019 leader_election.cc:288] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [CANDIDATE]: Term 1 election: Requested vote from peers 35f0f315077c4415a8ae85cb40f855b8 (127.0.98.1:46555), 33d94adc8a5f4dc1820bdf4c94d0f77f (127.0.98.2:38645)
I20260926 01:04:48.167394 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71592338ba544d248d9df5251f950f9d" candidate_uuid: "9c254457ba5a4f3e8907589375637145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f0f315077c4415a8ae85cb40f855b8"
I20260926 01:04:48.167497 1663 raft_consensus.cc:3037] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:48.167469 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71592338ba544d248d9df5251f950f9d" candidate_uuid: "9c254457ba5a4f3e8907589375637145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f"
I20260926 01:04:48.167536 1804 raft_consensus.cc:3037] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:48.168087 1804 raft_consensus.cc:2454] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c254457ba5a4f3e8907589375637145 in term 1.
I20260926 01:04:48.168186 1663 raft_consensus.cc:2454] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c254457ba5a4f3e8907589375637145 in term 1.
I20260926 01:04:48.168285 1875 leader_election.cc:302] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d94adc8a5f4dc1820bdf4c94d0f77f, 9c254457ba5a4f3e8907589375637145; no voters:
I20260926 01:04:48.168380 2019 raft_consensus.cc:2781] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:48.168433 2019 raft_consensus.cc:686] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [term 1 LEADER]: Becoming Leader. State: Replica: 9c254457ba5a4f3e8907589375637145, State: Running, Role: LEADER
I20260926 01:04:48.168483 2019 consensus_queue.cc:237] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } }
I20260926 01:04:48.168859 2043 tablet_replica.cc:439] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: TxnStatusTablet state changed. Reason: New leader 9c254457ba5a4f3e8907589375637145. Latest consensus state: current_term: 1 leader_uuid: "9c254457ba5a4f3e8907589375637145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } }
I20260926 01:04:48.168952 2043 tablet_replica.cc:442] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:04:48.169104 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 01:04:48.169175 1507 catalog_manager.cc:5495] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c254457ba5a4f3e8907589375637145 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c254457ba5a4f3e8907589375637145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } health_report { overall_health: HEALTHY } } }
I20260926 01:04:48.169272 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 01:04:48.240155 2043 consensus_queue.cc:1035] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:48.240825 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:48.237705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40353 (local address 127.0.0.1:54630)
0926 01:04:48.237778 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:48.237793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:48.237840 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:48.238066 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:48.238067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:48.238069 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:04:48.238119 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:48.238121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.239089 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.239090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:48.239769 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:48.239772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.240202 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.240204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:48.240269 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:48.240291 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:04:48.240621 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:04:48.240652 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 01:04:48.240691 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 01:04:48.240750 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:04:48.241106 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:48.237801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.0.98.3:40353)
0926 01:04:48.237864 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:48.237866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:48.237899 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:04:48.237934 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:48.237935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:48.237955 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:48.238032 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:48.238202 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.238996 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.239845 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.240130 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.240206 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:48.240404 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:48.240531 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:48.240595 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:48.241019 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 01:04:48.241042 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:04:48.242136 1663 raft_consensus.cc:1261] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8 [term 1 FOLLOWER]: Refusing update from remote peer 9c254457ba5a4f3e8907589375637145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:48.242430 1804 raft_consensus.cc:1261] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 1 FOLLOWER]: Refusing update from remote peer 9c254457ba5a4f3e8907589375637145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:48.242682 2019 consensus_queue.cc:1035] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:48.242805 2019 consensus_queue.cc:1035] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:48.242866 2019 consensus_queue.cc:1035] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:48.243150 2021 tablet_replica.cc:439] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: TxnStatusTablet state changed. Reason: New leader 9c254457ba5a4f3e8907589375637145. Latest consensus state: current_term: 1 leader_uuid: "9c254457ba5a4f3e8907589375637145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } }
I20260926 01:04:48.243248 2021 tablet_replica.cc:442] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:48.243613 2023 tablet_replica.cc:439] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: TxnStatusTablet state changed. Reason: New leader 9c254457ba5a4f3e8907589375637145. Latest consensus state: current_term: 1 leader_uuid: "9c254457ba5a4f3e8907589375637145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } }
I20260926 01:04:48.243721 2023 tablet_replica.cc:442] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:48.244383 2043 tablet_replica.cc:439] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c254457ba5a4f3e8907589375637145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } }
I20260926 01:04:48.244472 2043 tablet_replica.cc:442] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:04:48.244938 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333415683037609984 to 7333415682738888704. Current Snapshot: MvccSnapshot[applied={T|T < 7333415683037609984}]
I20260926 01:04:48.245042 2021 tablet_replica.cc:439] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c254457ba5a4f3e8907589375637145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } }
I20260926 01:04:48.245131 2021 tablet_replica.cc:442] T 71592338ba544d248d9df5251f950f9d P 33d94adc8a5f4dc1820bdf4c94d0f77f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:48.245880 2019 tablet_replica.cc:439] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c254457ba5a4f3e8907589375637145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } }
I20260926 01:04:48.245982 2019 tablet_replica.cc:442] T 71592338ba544d248d9df5251f950f9d P 9c254457ba5a4f3e8907589375637145: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:04:48.246323 2018 tablet_replica.cc:439] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c254457ba5a4f3e8907589375637145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0f315077c4415a8ae85cb40f855b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46555 } } peers { permanent_uuid: "33d94adc8a5f4dc1820bdf4c94d0f77f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "9c254457ba5a4f3e8907589375637145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40353 } } }
I20260926 01:04:48.246438 2018 tablet_replica.cc:442] T 71592338ba544d248d9df5251f950f9d P 35f0f315077c4415a8ae85cb40f855b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:48.248714 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:48.246551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.0.98.3:40353)
0926 01:04:48.246593 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:48.246595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:48.246602 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:04:48.246623 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:48.246624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:48.246641 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:48.246687 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:48.246846 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.247357 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.248118 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.248319 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.248373 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:48.248501 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:48.248535 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:48.248578 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:48.248625 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 01:04:48.248646 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:04:48.253448 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:48.251173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59753 (local address 127.0.98.3:40353)
0926 01:04:48.251241 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:48.251243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:48.251340 (+ 97us) server_negotiation.cc:407] Connection header received
0926 01:04:48.251388 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:48.251389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:48.251399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:48.251442 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:48.251577 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.252095 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.252849 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.253024 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.253076 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:48.253216 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:48.253248 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:48.253287 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:48.253333 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 01:04:48.253353 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":35}
I20260926 01:04:48.253468 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:48.251117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40353 (local address 127.0.98.3:59753)
0926 01:04:48.251279 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:48.251289 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:48.251346 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:48.251442 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:48.251442 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:48.251444 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:04:48.251518 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:48.251520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.252164 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.252165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:48.252776 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:48.252777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.253078 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.253079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:48.253135 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:48.253153 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:04:48.253292 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:04:48.253293 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:04:48.253333 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 01:04:48.253371 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260926 01:04:48.267360 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:48.267504 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:48.268260 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dcf7ef7565040bd8b1930fcafa01379: Sending DeleteTablet for 3 replicas of tablet 09e74c4f47e843129aee53c0e14f8cc8
I20260926 01:04:48.268532 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 09e74c4f47e843129aee53c0e14f8cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42456
I20260926 01:04:48.268639 2037 tablet_replica.cc:330] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8: stopping tablet replica
I20260926 01:04:48.268702 2037 raft_consensus.cc:2229] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:48.268683 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 09e74c4f47e843129aee53c0e14f8cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47254
I20260926 01:04:48.268738 2037 raft_consensus.cc:2258] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:48.268811 2036 tablet_replica.cc:330] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f: stopping tablet replica
I20260926 01:04:48.268806 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 09e74c4f47e843129aee53c0e14f8cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54624
I20260926 01:04:48.268864 2036 raft_consensus.cc:2229] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:48.268885 2038 tablet_replica.cc:330] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: stopping tablet replica
I20260926 01:04:48.268900 2036 raft_consensus.cc:2258] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:48.268924 2038 raft_consensus.cc:2229] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:04:48.269019 2038 raft_consensus.cc:2258] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:48.269106 2037 ts_tablet_manager.cc:1905] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:48.269171 2036 ts_tablet_manager.cc:1905] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:48.269363 2038 ts_tablet_manager.cc:1905] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:48.271060 2037 ts_tablet_manager.cc:1918] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:04:48.271024 2036 ts_tablet_manager.cc:1918] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:04:48.271124 2037 log.cc:1196] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/09e74c4f47e843129aee53c0e14f8cc8
I20260926 01:04:48.271129 2036 log.cc:1196] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/09e74c4f47e843129aee53c0e14f8cc8
I20260926 01:04:48.271343 2037 ts_tablet_manager.cc:1939] T 09e74c4f47e843129aee53c0e14f8cc8 P 35f0f315077c4415a8ae85cb40f855b8: Deleting consensus metadata
I20260926 01:04:48.271343 2036 ts_tablet_manager.cc:1939] T 09e74c4f47e843129aee53c0e14f8cc8 P 33d94adc8a5f4dc1820bdf4c94d0f77f: Deleting consensus metadata
I20260926 01:04:48.271719 1491 catalog_manager.cc:4841] TS 33d94adc8a5f4dc1820bdf4c94d0f77f (127.0.98.2:38645): tablet 09e74c4f47e843129aee53c0e14f8cc8 (table test-table [id=550825db63b94473b9a1dd474f00e4ee]) successfully deleted
I20260926 01:04:48.271893 1493 catalog_manager.cc:4841] TS 35f0f315077c4415a8ae85cb40f855b8 (127.0.98.1:46555): tablet 09e74c4f47e843129aee53c0e14f8cc8 (table test-table [id=550825db63b94473b9a1dd474f00e4ee]) successfully deleted
I20260926 01:04:48.271970 2038 ts_tablet_manager.cc:1918] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:04:48.272028 2038 log.cc:1196] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/09e74c4f47e843129aee53c0e14f8cc8
I20260926 01:04:48.272240 2038 ts_tablet_manager.cc:1939] T 09e74c4f47e843129aee53c0e14f8cc8 P 9c254457ba5a4f3e8907589375637145: Deleting consensus metadata
I20260926 01:04:48.272570 1493 catalog_manager.cc:4841] TS 9c254457ba5a4f3e8907589375637145 (127.0.98.3:40353): tablet 09e74c4f47e843129aee53c0e14f8cc8 (table test-table [id=550825db63b94473b9a1dd474f00e4ee]) successfully deleted
Sep 26 01:04:48 dist-test-slave-l73v krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:04:48.280351 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:48.271644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.0.98.1:46555)
0926 01:04:48.271705 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:48.271707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:48.271719 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:04:48.271743 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:48.271744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:48.271755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:48.271824 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:48.272646 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.273194 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.273932 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.274188 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.274266 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:48.277217 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:48.277230 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:48.277230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:48.277238 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:48.278866 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:48.279328 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:48.279332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:48.279332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:48.279397 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:48.279727 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:48.279728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:48.279729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:48.279928 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:48.280051 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:48.280167 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 01:04:48.280271 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 01:04:48.280575 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42470
I20260926 01:04:48.288116 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:48.281437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.0.98.62:39421)
0926 01:04:48.281490 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:48.281491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:48.281656 (+ 165us) server_negotiation.cc:407] Connection header received
0926 01:04:48.281703 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:48.281704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:48.281714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:48.281760 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:48.282245 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.282811 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.283594 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:48.283806 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:48.283864 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:48.285151 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:48.285152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:48.285153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:48.285161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:48.286867 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:48.287258 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:48.287259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:48.287259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:48.287317 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:48.287507 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:48.287508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:48.287508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:48.287642 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:48.287722 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:48.287935 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 01:04:48.288019 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 01:04:48.288812 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45142
I20260926 01:04:48.289515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 1579
I20260926 01:04:48.297137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 1719
I20260926 01:04:48.301981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 1859
I20260926 01:04:48.307083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 1475
2026-09-26T01:04:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4084 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:04:48 dist-test-slave-l73v krb5kdc[2076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:04:48 dist-test-slave-l73v krb5kdc[2076](info): set up 2 sockets
Sep 26 01:04:48 dist-test-slave-l73v krb5kdc[2076](info): commencing operation
krb5kdc: starting...
W20260926 01:04:49.265993 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:04:49 dist-test-slave-l73v krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:04:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:04:49.387511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39833
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:49.470029 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:49.470259 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:49.470283 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:49.471565 2089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:04:49.471618 2089 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 01:04:49.471630 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:49.471642 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:49.471654 2089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:04:49.471664 2089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:04:49.473230 2089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39833
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:49.473480 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:49.473738 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:49.478709 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:49.478732 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:49.478916 2089 server_base.cc:1034] running on GCE node
W20260926 01:04:49.478740 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:49.479233 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:49.479525 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:49.480695 2089 hybrid_clock.cc:648] HybridClock initialized: now 1790384689480680 us; error 32 us; skew 500 ppm
Sep 26 01:04:49 dist-test-slave-l73v krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:49.483259 2089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:04:49.483757 2089 webserver.cc:460] Webserver started at http://127.0.98.62:44403/ using document root <none> and password file <none>
I20260926 01:04:49.483979 2089 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:49.484028 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:49.484141 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:49.485246 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "630a8784127349ff9f7cc66d13d6ec24"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "a491516c4b7a32012a85cb0e14462b2f"
server_key_iv: "b18aff1750e254f89759b3e711db5cb9"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.485601 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "630a8784127349ff9f7cc66d13d6ec24"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "a491516c4b7a32012a85cb0e14462b2f"
server_key_iv: "b18aff1750e254f89759b3e711db5cb9"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.487046 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:04:49.487759 2104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:49.487911 2089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:04:49.488013 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "630a8784127349ff9f7cc66d13d6ec24"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "a491516c4b7a32012a85cb0e14462b2f"
server_key_iv: "b18aff1750e254f89759b3e711db5cb9"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.488080 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:49.500303 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:49.502595 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:49.502731 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:49.507328 2089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39833
I20260926 01:04:49.507380 2166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39833 every 8 connection(s)
I20260926 01:04:49.507716 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:04:49.508335 2167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:49.510751 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24: Bootstrap starting.
I20260926 01:04:49.511345 2167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:49.511615 2167 log.cc:824] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:49.512441 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24: No bootstrap required, opened a new log
I20260926 01:04:49.512796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 2089
I20260926 01:04:49.512902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:04:49.513089 392 external_mini_cluster.cc:1428] Setting key 8ebb7b466150182b00afe1243e6c0105
I20260926 01:04:49.515301 2167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630a8784127349ff9f7cc66d13d6ec24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39833 } }
I20260926 01:04:49.515476 2167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:49.515512 2167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630a8784127349ff9f7cc66d13d6ec24, State: Initialized, Role: FOLLOWER
I20260926 01:04:49.515654 2167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630a8784127349ff9f7cc66d13d6ec24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39833 } }
I20260926 01:04:49.515728 2167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:04:49.515759 2167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:04:49.515811 2167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:49.516440 2167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630a8784127349ff9f7cc66d13d6ec24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39833 } }
I20260926 01:04:49.516547 2167 leader_election.cc:302] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 630a8784127349ff9f7cc66d13d6ec24; no voters:
I20260926 01:04:49.516736 2167 leader_election.cc:288] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:04:49.516776 2172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:49.516916 2172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [term 1 LEADER]: Becoming Leader. State: Replica: 630a8784127349ff9f7cc66d13d6ec24, State: Running, Role: LEADER
I20260926 01:04:49.517012 2167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:04:49.517117 2172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630a8784127349ff9f7cc66d13d6ec24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39833 } }
I20260926 01:04:49.518476 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "630a8784127349ff9f7cc66d13d6ec24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630a8784127349ff9f7cc66d13d6ec24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39833 } } }
I20260926 01:04:49.519450 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:49.518471 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 630a8784127349ff9f7cc66d13d6ec24. Latest consensus state: current_term: 1 leader_uuid: "630a8784127349ff9f7cc66d13d6ec24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630a8784127349ff9f7cc66d13d6ec24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39833 } } }
I20260926 01:04:49.520303 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24 [sys.catalog]: This master's current role is: LEADER
W20260926 01:04:49.518952 2187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:04:49.520421 2187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:04:49.520503 2189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:04:49.520709 2189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 01:04:49 dist-test-slave-l73v krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:49.523175 2189 catalog_manager.cc:1347] Generated new cluster ID: 107d174226314979a44ed44d24853879
I20260926 01:04:49.523226 2189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:04:49.524961 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:49.513721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.0.98.62:39833)
0926 01:04:49.513900 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:49.513908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:49.513933 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:04:49.515212 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:49.515221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:49.515278 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:49.515424 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:49.516112 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.517251 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.518308 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.518833 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.518943 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:49.521410 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:49.521414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:49.521414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:49.521428 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:49.523153 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:49.523555 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:49.523558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:49.523559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:49.523656 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:49.523812 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:49.523812 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:49.523813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:49.524094 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:49.524223 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:49.524632 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 01:04:49.524763 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260926 01:04:49.539355 2189 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:04:49.540112 2189 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:04:49.549371 2189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24: Generated new TSK 0
I20260926 01:04:49.549602 2189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:04:49.578450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39833
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:49.659937 2193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:49.660171 2193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:49.660197 2193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:49.661402 2193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:49.661446 2193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:49.661466 2193 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 01:04:49.661477 2193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:04:49.661494 2193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:04:49.663081 2193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:49.663312 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:49.663548 2193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:49.668694 2199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:49.668718 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:49.668718 2201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:49.669487 2193 server_base.cc:1034] running on GCE node
I20260926 01:04:49.669687 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:49.669927 2193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:49.671082 2193 hybrid_clock.cc:648] HybridClock initialized: now 1790384689671066 us; error 31 us; skew 500 ppm
Sep 26 01:04:49 dist-test-slave-l73v krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:49.673786 2193 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:04:49.674237 2193 webserver.cc:460] Webserver started at http://127.0.98.1:39001/ using document root <none> and password file <none>
I20260926 01:04:49.674475 2193 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:49.674531 2193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:49.674636 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:49.675722 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "ad787bf0720adf877e28affc1277b346"
server_key_iv: "f03339f5a2fa7cc1071954bbdd28edfa"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.676030 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "ad787bf0720adf877e28affc1277b346"
server_key_iv: "f03339f5a2fa7cc1071954bbdd28edfa"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.677250 2193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:04:49.678063 2208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:49.678304 2193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:04:49.678433 2193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "ad787bf0720adf877e28affc1277b346"
server_key_iv: "f03339f5a2fa7cc1071954bbdd28edfa"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.678511 2193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:49.686820 2193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:49.689246 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:49.689445 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:49.689987 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:49.690032 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:49.690064 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:49.690089 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:49.696561 2193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38025
I20260926 01:04:49.696746 2323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38025 every 8 connection(s)
I20260926 01:04:49.696996 2193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:04:49 dist-test-slave-l73v krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:49.701103 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:49.690523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38423 (local address 127.0.98.62:39833)
0926 01:04:49.690576 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:49.690578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:49.690981 (+ 403us) server_negotiation.cc:407] Connection header received
0926 01:04:49.691640 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:49.691641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:49.691654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:49.691701 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:49.692563 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.693396 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.694970 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.695198 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.695262 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:49.698056 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:49.698058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:49.698058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:49.698066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:49.699411 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:49.699933 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:49.699934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:49.699935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:49.700000 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:49.700326 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:49.700327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:49.700328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:49.700469 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:49.700539 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:49.700890 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 01:04:49.700979 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:04:49.701180 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:49.690545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39833 (local address 127.0.98.1:38423)
0926 01:04:49.690834 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:49.690885 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:49.691500 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:49.691782 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:49.691788 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:49.692162 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:49.692471 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:49.692479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.693494 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.693495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:49.694879 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:49.694883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.695291 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.695292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:49.695469 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:49.696174 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:49.696185 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:49.697956 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:49.699513 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:49.699521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:49.699523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:49.699808 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:49.700086 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:49.700087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:49.700088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:49.700209 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:49.700556 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:49.700562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:49.700704 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 01:04:49.700822 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 01:04:49.700964 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":123,"threads_started":1}
I20260926 01:04:49.701733 2324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39833
I20260926 01:04:49.701835 2324 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:49.702072 2324 heartbeater.cc:511] Master 127.0.98.62:39833 requested a full tablet report, sending...
I20260926 01:04:49.702663 2119 ts_manager.cc:194] Registered new tserver with Master: 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025)
I20260926 01:04:49.703323 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38423
I20260926 01:04:49.703595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 2193
I20260926 01:04:49.703693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:04:49.703900 392 external_mini_cluster.cc:1428] Setting key 875251da5820f5ad540285d6385d996c
I20260926 01:04:49.708958 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:49.702905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45471 (local address 127.0.98.62:39833)
0926 01:04:49.702984 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:49.702986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:49.703107 (+ 121us) server_negotiation.cc:407] Connection header received
0926 01:04:49.703160 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:49.703161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:49.703171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:49.703216 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:49.703570 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.704267 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.705053 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.705321 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.705381 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:49.706461 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:49.706462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:49.706462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:49.706470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:49.707775 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:49.708184 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:49.708185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:49.708185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:49.708240 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:49.708459 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:49.708459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:49.708460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:49.708563 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:49.708611 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:49.708805 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 01:04:49.708892 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 01:04:49.708984 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:49.702852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39833 (local address 127.0.98.1:45471)
0926 01:04:49.703039 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:49.703048 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:49.703105 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:49.703219 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:49.703220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:49.703394 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:49.703474 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:49.703476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.704389 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.704390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:49.704972 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:49.704975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.705388 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.705389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:49.705508 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:49.705938 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:49.705943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:49.706330 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:49.707865 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:49.707866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:49.707867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:49.708090 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:49.708299 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:49.708300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:49.708300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:49.708368 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:49.708618 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:49.708619 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:49.708674 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 01:04:49.708750 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 01:04:49.708867 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:04:49.730527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39833
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:49.813757 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:49.813997 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:49.814023 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:49.815237 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:49.815275 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:49.815295 2333 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 01:04:49.815307 2333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:04:49.815323 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:04:49.816723 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:49.816924 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:49.817118 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:49.822118 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:49.822154 2338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:49.822134 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:49.822171 2333 server_base.cc:1034] running on GCE node
I20260926 01:04:49.822553 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:49.822788 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:49.823959 2333 hybrid_clock.cc:648] HybridClock initialized: now 1790384689823943 us; error 31 us; skew 500 ppm
Sep 26 01:04:49 dist-test-slave-l73v krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:49.826543 2333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:04:49.827011 2333 webserver.cc:460] Webserver started at http://127.0.98.2:39173/ using document root <none> and password file <none>
I20260926 01:04:49.827219 2333 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:49.827266 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:49.827394 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:49.828464 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "9e73d76125374f7ebdb69916c2bb2ad4"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "1cf85dab1d2934c33571773d38b83693"
server_key_iv: "984c4b62bd7082e6ebc1d8942e08eb31"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.828804 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9e73d76125374f7ebdb69916c2bb2ad4"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "1cf85dab1d2934c33571773d38b83693"
server_key_iv: "984c4b62bd7082e6ebc1d8942e08eb31"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.830109 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:04:49.831004 2348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:49.831197 2333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:04:49.831290 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "9e73d76125374f7ebdb69916c2bb2ad4"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "1cf85dab1d2934c33571773d38b83693"
server_key_iv: "984c4b62bd7082e6ebc1d8942e08eb31"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.831372 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:49.846297 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:49.848414 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:49.848605 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:49.849121 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:49.849162 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:49.849198 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:49.849220 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:49.856024 2333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44401
I20260926 01:04:49.856089 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44401 every 8 connection(s)
I20260926 01:04:49.856397 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:04:49 dist-test-slave-l73v krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:49.859848 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:49.849706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44373 (local address 127.0.98.62:39833)
0926 01:04:49.849760 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:49.849762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:49.850022 (+ 260us) server_negotiation.cc:407] Connection header received
0926 01:04:49.850594 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:49.850596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:49.850610 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:49.850683 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:49.851613 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.852494 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.854096 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.854348 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.854456 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:49.857102 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:49.857104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:49.857104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:49.857112 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:49.858316 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:49.858812 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:49.858812 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:49.858813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:49.858866 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:49.859113 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:49.859114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:49.859114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:49.859313 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:49.859388 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:49.859704 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 01:04:49.859780 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:04:49.859990 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:49.849723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39833 (local address 127.0.98.2:44373)
0926 01:04:49.849899 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:49.849939 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:49.850491 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:49.850756 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:49.850760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:49.851192 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:49.851499 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:49.851507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.852594 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.852595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:49.853997 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:49.854002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.854464 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.854465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:49.854607 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:49.855272 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:49.855285 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:49.856983 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:49.858493 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:49.858499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:49.858502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:49.858736 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:49.858919 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:49.858919 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:49.858919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:49.859017 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:49.859387 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:49.859423 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:49.859556 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 01:04:49.859647 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 01:04:49.859790 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260926 01:04:49.860473 2464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39833
I20260926 01:04:49.860564 2464 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:49.860741 2464 heartbeater.cc:511] Master 127.0.98.62:39833 requested a full tablet report, sending...
I20260926 01:04:49.861135 2121 ts_manager.cc:194] Registered new tserver with Master: 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401)
I20260926 01:04:49.861578 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44373
I20260926 01:04:49.865648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 2333
I20260926 01:04:49.865741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:04:49.865934 392 external_mini_cluster.cc:1428] Setting key 36d2778137031ee91f5b5d1712921cb9
I20260926 01:04:49.867379 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:49.861515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55373 (local address 127.0.98.62:39833)
0926 01:04:49.861555 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:49.861557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:49.861655 (+ 98us) server_negotiation.cc:407] Connection header received
0926 01:04:49.861728 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:49.861729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:49.861739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:49.861789 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:49.862111 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.862657 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.863325 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.863522 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.863577 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:49.864537 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:49.864539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:49.864539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:49.864547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:49.865999 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:49.866477 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:49.866478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:49.866479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:49.866548 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:49.866744 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:49.866745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:49.866746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:49.866952 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:49.867011 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:49.867240 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 01:04:49.867316 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:04:49.867434 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:49.861464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39833 (local address 127.0.98.2:55373)
0926 01:04:49.861598 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:49.861607 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:49.861678 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:49.861791 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:49.861792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:49.861981 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:49.862037 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:49.862039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.862748 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.862749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:49.863253 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:49.863255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:49.863582 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:49.863583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:49.863656 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:49.864097 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:49.864102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:49.864439 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:49.866106 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:49.866107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:49.866108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:49.866354 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:49.866620 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:49.866621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:49.866621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:49.866679 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:49.867043 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:49.867044 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:49.867102 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 01:04:49.867185 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 01:04:49.867304 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:04:49.893863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39833
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:49.973345 2473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:49.973585 2473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:49.973623 2473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:49.974920 2473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:49.974974 2473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:49.975001 2473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 01:04:49.975016 2473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:04:49.975040 2473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:04:49.976557 2473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:49.976799 2473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:49.977015 2473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:49.982124 2478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:49.982121 2479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:49.982152 2481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:49.982553 2473 server_base.cc:1034] running on GCE node
I20260926 01:04:49.982759 2473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:49.982998 2473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:49.984157 2473 hybrid_clock.cc:648] HybridClock initialized: now 1790384689984137 us; error 33 us; skew 500 ppm
Sep 26 01:04:49 dist-test-slave-l73v krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:49.986877 2473 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:04:49.987329 2473 webserver.cc:460] Webserver started at http://127.0.98.3:34541/ using document root <none> and password file <none>
I20260926 01:04:49.987530 2473 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:49.987578 2473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:49.987679 2473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:49.988757 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b76215cc14f457eb2c1497d54b02e0c"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "693cd016a032dbcb9bb8db12f3639c12"
server_key_iv: "37a53018c32c6f0e28b424a1c893d9da"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.989061 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b76215cc14f457eb2c1497d54b02e0c"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "693cd016a032dbcb9bb8db12f3639c12"
server_key_iv: "37a53018c32c6f0e28b424a1c893d9da"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.990476 2473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:04:49.991291 2488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:49.991433 2473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:04:49.991549 2473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "3b76215cc14f457eb2c1497d54b02e0c"
format_stamp: "Formatted at 2026-09-26 01:04:49 on dist-test-slave-l73v"
server_key: "693cd016a032dbcb9bb8db12f3639c12"
server_key_iv: "37a53018c32c6f0e28b424a1c893d9da"
server_key_version: "encryptionkey@0"
I20260926 01:04:49.991611 2473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:49.999657 2473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:50.002310 2473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:50.002514 2473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:50.003036 2473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:50.003078 2473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:50.003113 2473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:50.003136 2473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:50.009325 2473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34471
I20260926 01:04:50.009465 2603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34471 every 8 connection(s)
I20260926 01:04:50.009730 2473 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:04:50 dist-test-slave-l73v krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:50.014223 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.003601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54925 (local address 127.0.98.62:39833)
0926 01:04:50.003657 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.003659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.003845 (+ 186us) server_negotiation.cc:407] Connection header received
0926 01:04:50.004438 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.004439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.004457 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.004566 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:50.005605 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.006373 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.007894 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.008198 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.008269 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.011036 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:50.011038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:50.011038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:50.011049 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:50.012657 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:50.013103 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:50.013105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:50.013105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:50.013170 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:50.013402 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:50.013402 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:50.013403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:50.013567 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:50.013729 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.014028 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.014124 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:04:50.014518 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.003548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39833 (local address 127.0.98.3:54925)
0926 01:04:50.003724 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.003764 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.004278 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.004691 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.004697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.005148 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:50.005463 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.005471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.006516 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.006517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.007793 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.007796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.008362 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.008363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.008533 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.009142 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:50.009154 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:50.010858 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:50.012750 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:50.012760 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:50.012762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:50.013016 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:50.013228 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:50.013228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:50.013229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:50.013342 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:50.013715 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:50.013723 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:50.013891 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.013994 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.014178 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260926 01:04:50.015100 2604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39833
I20260926 01:04:50.015199 2604 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:50.015381 2604 heartbeater.cc:511] Master 127.0.98.62:39833 requested a full tablet report, sending...
I20260926 01:04:50.015780 2119 ts_manager.cc:194] Registered new tserver with Master: 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3:34471)
I20260926 01:04:50.016263 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54925
I20260926 01:04:50.018599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 2473
I20260926 01:04:50.018692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:04:50.018888 392 external_mini_cluster.cc:1428] Setting key 4316fa3c8a18f1e1b192f138d949b638
I20260926 01:04:50.019752 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:04:50.022130 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.016303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35441 (local address 127.0.98.62:39833)
0926 01:04:50.016358 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.016360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.016480 (+ 120us) server_negotiation.cc:407] Connection header received
0926 01:04:50.016562 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.016563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.016571 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.016618 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:50.017042 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.017570 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.018366 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.018637 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.018707 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.019836 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:50.019837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:50.019838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:50.019845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:50.021038 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:50.021398 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:50.021400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:50.021400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:50.021475 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:50.021663 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:50.021664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:50.021664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:50.021765 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:50.021811 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.021989 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.022065 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:04:50.022154 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.016219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39833 (local address 127.0.98.3:35441)
0926 01:04:50.016415 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.016427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.016496 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.016637 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.016638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.016859 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:50.016927 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.016929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.017645 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.017646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.018269 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.018272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.018765 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.018766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.018870 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.019358 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:50.019363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:50.019771 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:50.021121 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:50.021122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:50.021123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:50.021303 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:50.021542 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:50.021543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:50.021544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:50.021611 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:50.021814 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:50.021815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:50.021866 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.021930 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.022048 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":76,"threads_started":1}
Sep 26 01:04:50 dist-test-slave-l73v krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:04:50 dist-test-slave-l73v krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:50.044112 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.036958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.98.62:39833)
0926 01:04:50.037044 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.037046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.037056 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:04:50.037081 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.037081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.037090 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.037144 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:50.037718 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.038227 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.039100 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.039289 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.039344 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.041583 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:50.041584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:50.041585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:50.041592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:50.043012 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:50.043382 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:50.043383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:50.043383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:50.043435 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:50.043640 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:50.043640 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:50.043640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:50.043742 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:50.043811 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.043981 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.044043 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 01:04:50.045917 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:04:50.047617 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:04:50.054056 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.050052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.0.98.3:34471)
0926 01:04:50.050229 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.050234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.050241 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:04:50.050272 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.050274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.050287 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.050354 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:50.050641 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.051793 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.053038 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.053601 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.053667 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.053734 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.053902 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.053948 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260926 01:04:50.054039 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.049759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34471 (local address 127.0.0.1:47682)
0926 01:04:50.050041 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.050051 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.050110 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.050403 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.050407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.050410 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:50.050564 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.050566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.051896 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.051897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.052952 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.052953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.053688 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.053689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.053783 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.053809 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.053884 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.053944 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260926 01:04:50.054129 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.049859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.0.98.2:44401)
0926 01:04:50.050558 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.050562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.050570 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:04:50.050602 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.050603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.050615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.050801 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:50.050857 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.051537 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.052984 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.053684 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.053784 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.053835 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.053998 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.054039 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":96,"threads_started":1}
I20260926 01:04:50.054211 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.050138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.0.98.1:38025)
0926 01:04:50.050296 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.050301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.050430 (+ 129us) server_negotiation.cc:407] Connection header received
0926 01:04:50.050498 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.050500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.050516 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.050589 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:50.050777 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.051901 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.052922 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.053684 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.053755 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.053815 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.054080 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.054122 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260926 01:04:50.054510 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.049759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44401 (local address 127.0.0.1:50836)
0926 01:04:50.049937 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.049981 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.050058 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.050703 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.050704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.050706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:50.050735 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.050736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.051618 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.051619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.052903 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.052906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.053789 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.053790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.053843 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.053846 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.053899 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.053944 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260926 01:04:50.054581 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.049959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38025 (local address 127.0.0.1:35436)
0926 01:04:50.050298 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.050347 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.050432 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.050656 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.050657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.050659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:50.050713 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.050715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.051961 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.051962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.052842 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.052845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.053848 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.053849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.053909 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.053914 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.053973 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.054015 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":38,"threads_started":1}
I20260926 01:04:50.055348 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 24384d60960046589f1715e328f53f94 (DEFAULT_TABLE table=test-table [id=9a4b7c99eb9749b0947195fc647e02cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:50.055367 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 24384d60960046589f1715e328f53f94 (DEFAULT_TABLE table=test-table [id=9a4b7c99eb9749b0947195fc647e02cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:50.055366 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 24384d60960046589f1715e328f53f94 (DEFAULT_TABLE table=test-table [id=9a4b7c99eb9749b0947195fc647e02cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:50.055706 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24384d60960046589f1715e328f53f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:50.055723 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24384d60960046589f1715e328f53f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:50.055723 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24384d60960046589f1715e328f53f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:50.058269 2624 tablet_bootstrap.cc:492] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: Bootstrap starting.
I20260926 01:04:50.059150 2624 tablet_bootstrap.cc:654] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:50.059437 2624 log.cc:824] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:50.059726 2626 tablet_bootstrap.cc:492] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Bootstrap starting.
I20260926 01:04:50.059756 2625 tablet_bootstrap.cc:492] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: Bootstrap starting.
I20260926 01:04:50.060392 2625 tablet_bootstrap.cc:654] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:50.060403 2626 tablet_bootstrap.cc:654] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:50.060539 2624 tablet_bootstrap.cc:492] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: No bootstrap required, opened a new log
I20260926 01:04:50.060626 2624 ts_tablet_manager.cc:1397] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:04:50.060726 2626 log.cc:824] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:50.060726 2625 log.cc:824] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:50.061475 2626 tablet_bootstrap.cc:492] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: No bootstrap required, opened a new log
I20260926 01:04:50.061481 2625 tablet_bootstrap.cc:492] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: No bootstrap required, opened a new log
I20260926 01:04:50.061546 2625 ts_tablet_manager.cc:1397] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:04:50.061546 2626 ts_tablet_manager.cc:1397] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:04:50.062311 2624 raft_consensus.cc:348] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260926 01:04:50.062511 2624 raft_consensus.cc:374] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:50.062552 2624 raft_consensus.cc:729] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b76215cc14f457eb2c1497d54b02e0c, State: Initialized, Role: FOLLOWER
I20260926 01:04:50.062672 2624 consensus_queue.cc:260] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260926 01:04:50.062958 2624 ts_tablet_manager.cc:1428] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:04:50.063076 2604 heartbeater.cc:503] Master 127.0.98.62:39833 was elected leader, sending a full tablet report...
I20260926 01:04:50.063742 2626 raft_consensus.cc:348] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260926 01:04:50.063742 2625 raft_consensus.cc:348] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260926 01:04:50.063936 2626 raft_consensus.cc:374] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:50.063942 2625 raft_consensus.cc:374] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:50.063974 2626 raft_consensus.cc:729] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9296e0d3ec3a4d01ac0271a2ceaaa66d, State: Initialized, Role: FOLLOWER
I20260926 01:04:50.063987 2625 raft_consensus.cc:729] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e73d76125374f7ebdb69916c2bb2ad4, State: Initialized, Role: FOLLOWER
I20260926 01:04:50.064096 2625 consensus_queue.cc:260] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260926 01:04:50.064096 2626 consensus_queue.cc:260] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260926 01:04:50.064381 2625 ts_tablet_manager.cc:1428] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:04:50.064383 2626 ts_tablet_manager.cc:1428] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:04:50.064411 2324 heartbeater.cc:503] Master 127.0.98.62:39833 was elected leader, sending a full tablet report...
I20260926 01:04:50.064508 2464 heartbeater.cc:503] Master 127.0.98.62:39833 was elected leader, sending a full tablet report...
W20260926 01:04:50.107043 2465 tablet.cc:2369] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:04:50.197826 2325 tablet.cc:2369] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:04:50.260434 2605 tablet.cc:2369] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:04:50.282542 2630 raft_consensus.cc:482] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:04:50.282691 2630 raft_consensus.cc:504] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260926 01:04:50.283052 2630 leader_election.cc:288] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025), 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401)
I20260926 01:04:50.285876 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.283263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32927 (local address 127.0.98.1:38025)
0926 01:04:50.283314 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.283316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.283328 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:04:50.283393 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.283394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.283409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.283458 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:50.283619 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.284139 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.285208 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.285569 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.285631 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.285642 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.285774 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.285812 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:04:50.285876 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.283332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59671 (local address 127.0.98.2:44401)
0926 01:04:50.283418 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.283420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.283479 (+ 59us) server_negotiation.cc:407] Connection header received
0926 01:04:50.283533 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.283534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.283550 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.283607 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:50.283776 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.284301 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.285191 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.285548 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.285602 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.285612 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.285758 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.285796 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:04:50.285879 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.283212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38025 (local address 127.0.98.3:32927)
0926 01:04:50.283266 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.283277 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.283317 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.283466 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.283467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.283469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:50.283545 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.283547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.284230 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.284231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.285109 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.285111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.285647 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.285648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.285707 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.285710 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.285757 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.285798 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":3}
I20260926 01:04:50.285887 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.283243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44401 (local address 127.0.98.3:59671)
0926 01:04:50.283411 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.283420 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.283474 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.283618 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.283619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.283620 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:50.283692 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.283693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.284380 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.284381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.285110 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.285112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.285624 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.285625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.285697 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.285701 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.285744 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.285783 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260926 01:04:50.286273 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24384d60960046589f1715e328f53f94" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" is_pre_election: true
I20260926 01:04:50.286273 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24384d60960046589f1715e328f53f94" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" is_pre_election: true
I20260926 01:04:50.286465 2278 raft_consensus.cc:2454] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 0.
I20260926 01:04:50.286465 2418 raft_consensus.cc:2454] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 0.
I20260926 01:04:50.286751 2490 leader_election.cc:302] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b76215cc14f457eb2c1497d54b02e0c, 9e73d76125374f7ebdb69916c2bb2ad4; no voters:
I20260926 01:04:50.286873 2630 raft_consensus.cc:2781] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:04:50.286940 2630 raft_consensus.cc:482] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:04:50.286965 2630 raft_consensus.cc:3037] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:50.287899 2630 raft_consensus.cc:504] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260926 01:04:50.288033 2630 leader_election.cc:288] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 election: Requested vote from peers 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025), 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401)
I20260926 01:04:50.288211 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24384d60960046589f1715e328f53f94" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73d76125374f7ebdb69916c2bb2ad4"
I20260926 01:04:50.288266 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24384d60960046589f1715e328f53f94" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d"
I20260926 01:04:50.288311 2418 raft_consensus.cc:3037] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:50.288335 2278 raft_consensus.cc:3037] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:50.289179 2278 raft_consensus.cc:2454] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 1.
I20260926 01:04:50.289179 2418 raft_consensus.cc:2454] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 1.
I20260926 01:04:50.289367 2490 leader_election.cc:302] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b76215cc14f457eb2c1497d54b02e0c, 9e73d76125374f7ebdb69916c2bb2ad4; no voters:
I20260926 01:04:50.289481 2630 raft_consensus.cc:2781] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:50.289637 2630 raft_consensus.cc:686] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 1 LEADER]: Becoming Leader. State: Replica: 3b76215cc14f457eb2c1497d54b02e0c, State: Running, Role: LEADER
I20260926 01:04:50.289726 2630 consensus_queue.cc:237] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260926 01:04:50.290583 2120 catalog_manager.cc:5495] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } health_report { overall_health: UNKNOWN } } }
Sep 26 01:04:50 dist-test-slave-l73v krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:50.363672 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.355366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39833 (local address 127.0.0.1:60574)
0926 01:04:50.355588 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.355598 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.355775 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.356001 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.356002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.356361 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:50.356412 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.356414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.357146 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.357146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.357721 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.357723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.358127 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.358128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.358182 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.358783 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:50.358788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:50.360764 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:50.362436 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:50.362447 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:50.362451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:50.362679 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:50.362937 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:50.362938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:50.362938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:50.363069 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:50.363351 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:50.363359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:50.363409 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.363464 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.363590 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
I20260926 01:04:50.363795 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.355475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.0.98.62:39833)
0926 01:04:50.355547 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.355549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.355737 (+ 188us) server_negotiation.cc:407] Connection header received
0926 01:04:50.355886 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.355887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.355897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.355950 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:50.356507 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.357051 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.357829 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.358025 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.358086 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.360936 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:50.360938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:50.360938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:50.360946 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:50.362301 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:50.362773 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:50.362775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:50.362775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:50.362837 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:50.363150 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:50.363151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:50.363152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:50.363279 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:50.363344 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.363535 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.363709 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":43}
I20260926 01:04:50.366693 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60574:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 01:04:50.366900 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:04:50.368921 2398 tablet_service.cc:1467] Processing CreateTablet for tablet df8bcd8fc66046aa846228195ebe972d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=485143f483144a4e8aa83ca105b4e3ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:04:50.368907 2258 tablet_service.cc:1467] Processing CreateTablet for tablet df8bcd8fc66046aa846228195ebe972d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=485143f483144a4e8aa83ca105b4e3ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:04:50.369076 2538 tablet_service.cc:1467] Processing CreateTablet for tablet df8bcd8fc66046aa846228195ebe972d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=485143f483144a4e8aa83ca105b4e3ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:04:50.369132 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8bcd8fc66046aa846228195ebe972d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:50.369153 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8bcd8fc66046aa846228195ebe972d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:50.369205 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8bcd8fc66046aa846228195ebe972d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:50.370450 2625 tablet_bootstrap.cc:492] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: Bootstrap starting.
I20260926 01:04:50.370673 2624 tablet_bootstrap.cc:492] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: Bootstrap starting.
I20260926 01:04:50.370887 2625 tablet_bootstrap.cc:654] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:50.371073 2624 tablet_bootstrap.cc:654] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:50.371415 2626 tablet_bootstrap.cc:492] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Bootstrap starting.
I20260926 01:04:50.371538 2625 tablet_bootstrap.cc:492] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: No bootstrap required, opened a new log
I20260926 01:04:50.371594 2625 ts_tablet_manager.cc:1397] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:50.371775 2626 tablet_bootstrap.cc:654] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:50.371755 2625 raft_consensus.cc:348] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.371819 2625 raft_consensus.cc:374] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:50.371835 2625 raft_consensus.cc:729] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e73d76125374f7ebdb69916c2bb2ad4, State: Initialized, Role: FOLLOWER
I20260926 01:04:50.371873 2625 consensus_queue.cc:260] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.371997 2625 ts_tablet_manager.cc:1428] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:50.372139 2632 tablet_replica.cc:439] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } }
I20260926 01:04:50.372236 2632 tablet_replica.cc:442] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:50.372453 2624 tablet_bootstrap.cc:492] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: No bootstrap required, opened a new log
I20260926 01:04:50.372503 2624 ts_tablet_manager.cc:1397] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:04:50.372653 2624 raft_consensus.cc:348] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.372740 2624 raft_consensus.cc:374] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:50.372762 2624 raft_consensus.cc:729] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b76215cc14f457eb2c1497d54b02e0c, State: Initialized, Role: FOLLOWER
I20260926 01:04:50.372802 2624 consensus_queue.cc:260] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.372906 2624 ts_tablet_manager.cc:1428] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:50.372980 2630 tablet_replica.cc:439] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } }
I20260926 01:04:50.373070 2630 tablet_replica.cc:442] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:50.373308 2626 tablet_bootstrap.cc:492] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: No bootstrap required, opened a new log
I20260926 01:04:50.373356 2626 ts_tablet_manager.cc:1397] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:04:50.373502 2626 raft_consensus.cc:348] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.373564 2626 raft_consensus.cc:374] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:50.373580 2626 raft_consensus.cc:729] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9296e0d3ec3a4d01ac0271a2ceaaa66d, State: Initialized, Role: FOLLOWER
I20260926 01:04:50.373618 2626 consensus_queue.cc:260] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.373687 2626 ts_tablet_manager.cc:1428] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:50.373752 2631 tablet_replica.cc:439] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } }
I20260926 01:04:50.373852 2631 tablet_replica.cc:442] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:50.440382 2630 raft_consensus.cc:482] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:04:50.440496 2630 raft_consensus.cc:504] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.440699 2630 leader_election.cc:288] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025), 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401)
I20260926 01:04:50.440948 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8bcd8fc66046aa846228195ebe972d" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" is_pre_election: true
I20260926 01:04:50.440956 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8bcd8fc66046aa846228195ebe972d" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" is_pre_election: true
I20260926 01:04:50.441033 2418 raft_consensus.cc:2454] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 0.
I20260926 01:04:50.441056 2278 raft_consensus.cc:2454] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 0.
I20260926 01:04:50.441184 2490 leader_election.cc:302] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b76215cc14f457eb2c1497d54b02e0c, 9e73d76125374f7ebdb69916c2bb2ad4; no voters:
I20260926 01:04:50.441288 2630 raft_consensus.cc:2781] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:04:50.441325 2630 raft_consensus.cc:482] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:04:50.441340 2630 raft_consensus.cc:3037] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:50.442018 2630 raft_consensus.cc:504] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.442142 2630 leader_election.cc:288] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 election: Requested vote from peers 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025), 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401)
I20260926 01:04:50.442334 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8bcd8fc66046aa846228195ebe972d" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73d76125374f7ebdb69916c2bb2ad4"
I20260926 01:04:50.442351 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8bcd8fc66046aa846228195ebe972d" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d"
I20260926 01:04:50.442442 2418 raft_consensus.cc:3037] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:50.442452 2278 raft_consensus.cc:3037] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:50.443066 2418 raft_consensus.cc:2454] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 1.
I20260926 01:04:50.443102 2278 raft_consensus.cc:2454] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 1.
I20260926 01:04:50.443246 2492 leader_election.cc:302] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b76215cc14f457eb2c1497d54b02e0c, 9296e0d3ec3a4d01ac0271a2ceaaa66d; no voters:
I20260926 01:04:50.443498 2630 raft_consensus.cc:2781] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:50.443619 2630 raft_consensus.cc:686] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [term 1 LEADER]: Becoming Leader. State: Replica: 3b76215cc14f457eb2c1497d54b02e0c, State: Running, Role: LEADER
I20260926 01:04:50.443791 2630 consensus_queue.cc:237] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.444186 2634 tablet_replica.cc:439] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: TxnStatusTablet state changed. Reason: New leader 3b76215cc14f457eb2c1497d54b02e0c. Latest consensus state: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } }
I20260926 01:04:50.444268 2634 tablet_replica.cc:442] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:04:50.444585 2121 catalog_manager.cc:5495] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } health_report { overall_health: HEALTHY } } }
I20260926 01:04:50.444841 2646 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 01:04:50.444960 2646 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 01:04:50.469519 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.466724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.0.98.3:34471)
0926 01:04:50.466792 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.466794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.466803 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:04:50.466897 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.466899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.466919 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.466977 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:50.467119 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.467609 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.468551 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.468770 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.468876 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.469182 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:50.469291 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:50.469369 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.469413 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.469435 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:04:50.469532 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.466635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34471 (local address 127.0.0.1:47696)
0926 01:04:50.466737 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.466750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.466808 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.467005 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.467006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.467008 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:04:50.467061 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.467064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.467693 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.467694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.468380 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.468384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.468898 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.468902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.468969 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.469048 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:04:50.469356 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:04:50.469364 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.469408 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.469465 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 01:04:50.471792 2418 raft_consensus.cc:1261] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Refusing update from remote peer 3b76215cc14f457eb2c1497d54b02e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:50.471781 2278 raft_consensus.cc:1261] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Refusing update from remote peer 3b76215cc14f457eb2c1497d54b02e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:50.472304 2634 consensus_queue.cc:1035] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:50.472464 2630 consensus_queue.cc:1035] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:50.473687 2631 tablet_replica.cc:439] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: TxnStatusTablet state changed. Reason: New leader 3b76215cc14f457eb2c1497d54b02e0c. Latest consensus state: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } }
I20260926 01:04:50.473686 2632 tablet_replica.cc:439] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: TxnStatusTablet state changed. Reason: New leader 3b76215cc14f457eb2c1497d54b02e0c. Latest consensus state: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } }
I20260926 01:04:50.473784 2632 tablet_replica.cc:442] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:50.473784 2631 tablet_replica.cc:442] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:50.474965 2634 tablet_replica.cc:439] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } }
I20260926 01:04:50.475070 2634 tablet_replica.cc:442] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:04:50.475148 2649 mvcc.cc:205] Tried to move back new op lower bound from 7333415692169428992 to 7333415692058161152. Current Snapshot: MvccSnapshot[applied={T|T < 7333415692169428992}]
I20260926 01:04:50.475411 2632 tablet_replica.cc:439] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } }
I20260926 01:04:50.475490 2632 tablet_replica.cc:442] T df8bcd8fc66046aa846228195ebe972d P 9e73d76125374f7ebdb69916c2bb2ad4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:50.475732 2651 mvcc.cc:205] Tried to move back new op lower bound from 7333415692169428992 to 7333415692058161152. Current Snapshot: MvccSnapshot[applied={T|T < 7333415692169428992}]
I20260926 01:04:50.476457 2630 tablet_replica.cc:439] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } }
I20260926 01:04:50.476589 2630 tablet_replica.cc:442] T df8bcd8fc66046aa846228195ebe972d P 3b76215cc14f457eb2c1497d54b02e0c: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:04:50.476814 2650 mvcc.cc:205] Tried to move back new op lower bound from 7333415692169428992 to 7333415692058161152. Current Snapshot: MvccSnapshot[applied={T|T < 7333415692169428992}]
I20260926 01:04:50.476976 2631 tablet_replica.cc:439] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } } }
I20260926 01:04:50.477062 2631 tablet_replica.cc:442] T df8bcd8fc66046aa846228195ebe972d P 9296e0d3ec3a4d01ac0271a2ceaaa66d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:04:50.480206 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.477906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47698 (local address 127.0.98.3:34471)
0926 01:04:50.477944 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.477946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.477954 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:04:50.478012 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.478012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.478022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.478067 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:50.478203 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.478769 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.479533 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.479733 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.479809 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.479973 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:50.480009 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:50.480059 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.480113 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.480134 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:04:50.485458 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.482912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33289 (local address 127.0.98.3:34471)
0926 01:04:50.482953 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.482955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.482961 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:04:50.482976 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.482977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.482985 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.483057 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:50.483196 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.483628 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.484711 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.484887 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.484940 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.485175 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:50.485217 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:50.485275 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.485341 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.485395 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:04:50.485520 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.482706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34471 (local address 127.0.98.3:33289)
0926 01:04:50.482774 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.482785 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.482830 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.483065 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.483066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.483067 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:04:50.483135 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.483137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.483689 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.483689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.484615 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.484618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.484951 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.484952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.485016 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.485036 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:04:50.485285 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:04:50.485286 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.485329 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.485380 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:04:50.487761 2278 raft_consensus.cc:1261] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Refusing update from remote peer 3b76215cc14f457eb2c1497d54b02e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:50.487761 2418 raft_consensus.cc:1261] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Refusing update from remote peer 3b76215cc14f457eb2c1497d54b02e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:50.487954 2634 consensus_queue.cc:1035] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:50.488122 2656 consensus_queue.cc:1035] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:50.499799 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:50.499904 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:50.500697 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24: Sending DeleteTablet for 3 replicas of tablet 24384d60960046589f1715e328f53f94
I20260926 01:04:50.501050 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 24384d60960046589f1715e328f53f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35436
I20260926 01:04:50.501025 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 24384d60960046589f1715e328f53f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50836
I20260926 01:04:50.501144 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 24384d60960046589f1715e328f53f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47682
I20260926 01:04:50.501472 2666 tablet_replica.cc:330] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: stopping tablet replica
I20260926 01:04:50.501458 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:04:50.501538 2664 tablet_replica.cc:330] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: stopping tablet replica
W20260926 01:04:50.501633 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:04:50.501658 2664 raft_consensus.cc:2229] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:50.501729 2664 raft_consensus.cc:2258] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:50.501866 2666 raft_consensus.cc:2229] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:04:50.501895 2665 tablet_replica.cc:330] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: stopping tablet replica
I20260926 01:04:50.501999 2665 raft_consensus.cc:2229] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:50.502063 2665 raft_consensus.cc:2258] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:50.502069 2666 raft_consensus.cc:2258] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:50.502739 2666 ts_tablet_manager.cc:1905] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:50.502970 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 3040604cdcca4e46a9e504a1d29aeb5a (DEFAULT_TABLE table=test-table [id=5cccf6492acf4fea8cbfd97820d7b648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:50.503110 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3040604cdcca4e46a9e504a1d29aeb5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:50.503094 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 3040604cdcca4e46a9e504a1d29aeb5a (DEFAULT_TABLE table=test-table [id=5cccf6492acf4fea8cbfd97820d7b648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:50.503180 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3040604cdcca4e46a9e504a1d29aeb5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:50.503173 2664 ts_tablet_manager.cc:1905] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:50.504710 2626 tablet_bootstrap.cc:492] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Bootstrap starting.
I20260926 01:04:50.504943 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 3040604cdcca4e46a9e504a1d29aeb5a (DEFAULT_TABLE table=test-table [id=5cccf6492acf4fea8cbfd97820d7b648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:50.505071 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3040604cdcca4e46a9e504a1d29aeb5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:50.505198 2665 ts_tablet_manager.cc:1905] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:50.505263 2626 tablet_bootstrap.cc:654] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:50.505625 2664 ts_tablet_manager.cc:1918] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:04:50.505704 2625 tablet_bootstrap.cc:492] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4: Bootstrap starting.
I20260926 01:04:50.506150 2664 log.cc:1196] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/24384d60960046589f1715e328f53f94
I20260926 01:04:50.506163 2625 tablet_bootstrap.cc:654] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:50.506781 2664 ts_tablet_manager.cc:1939] T 24384d60960046589f1715e328f53f94 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Deleting consensus metadata
I20260926 01:04:50.506815 2624 tablet_bootstrap.cc:492] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c: Bootstrap starting.
I20260926 01:04:50.507226 2624 tablet_bootstrap.cc:654] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:50.507287 2108 catalog_manager.cc:4841] TS 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025): tablet 24384d60960046589f1715e328f53f94 (table test-table [id=9a4b7c99eb9749b0947195fc647e02cd]) successfully deleted
I20260926 01:04:50.507690 2626 tablet_bootstrap.cc:492] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d: No bootstrap required, opened a new log
I20260926 01:04:50.507743 2626 ts_tablet_manager.cc:1397] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 01:04:50.507893 2626 raft_consensus.cc:348] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.507958 2626 raft_consensus.cc:374] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:50.507977 2626 raft_consensus.cc:729] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9296e0d3ec3a4d01ac0271a2ceaaa66d, State: Initialized, Role: FOLLOWER
I20260926 01:04:50.508018 2626 consensus_queue.cc:260] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.508097 2626 ts_tablet_manager.cc:1428] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:04:50.508816 2666 ts_tablet_manager.cc:1918] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:04:50.508898 2666 log.cc:1196] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/24384d60960046589f1715e328f53f94
I20260926 01:04:50.509150 2666 ts_tablet_manager.cc:1939] T 24384d60960046589f1715e328f53f94 P 3b76215cc14f457eb2c1497d54b02e0c: Deleting consensus metadata
I20260926 01:04:50.509384 2665 ts_tablet_manager.cc:1918] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:04:50.509465 2665 log.cc:1196] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/24384d60960046589f1715e328f53f94
I20260926 01:04:50.509701 2665 ts_tablet_manager.cc:1939] T 24384d60960046589f1715e328f53f94 P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting consensus metadata
I20260926 01:04:50.510269 2106 catalog_manager.cc:4841] TS 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401): tablet 24384d60960046589f1715e328f53f94 (table test-table [id=9a4b7c99eb9749b0947195fc647e02cd]) successfully deleted
I20260926 01:04:50.510597 2624 tablet_bootstrap.cc:492] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c: No bootstrap required, opened a new log
I20260926 01:04:50.510651 2624 ts_tablet_manager.cc:1397] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 01:04:50.510790 2624 raft_consensus.cc:348] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.510860 2624 raft_consensus.cc:374] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:50.510874 2624 raft_consensus.cc:729] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b76215cc14f457eb2c1497d54b02e0c, State: Initialized, Role: FOLLOWER
I20260926 01:04:50.510907 2624 consensus_queue.cc:260] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.511003 2624 ts_tablet_manager.cc:1428] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:50.511193 2108 catalog_manager.cc:4841] TS 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3:34471): tablet 24384d60960046589f1715e328f53f94 (table test-table [id=9a4b7c99eb9749b0947195fc647e02cd]) successfully deleted
I20260926 01:04:50.514942 2625 tablet_bootstrap.cc:492] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4: No bootstrap required, opened a new log
I20260926 01:04:50.515000 2625 ts_tablet_manager.cc:1397] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260926 01:04:50.515170 2625 raft_consensus.cc:348] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.515254 2625 raft_consensus.cc:374] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:50.515272 2625 raft_consensus.cc:729] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e73d76125374f7ebdb69916c2bb2ad4, State: Initialized, Role: FOLLOWER
I20260926 01:04:50.515316 2625 consensus_queue.cc:260] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.515403 2625 ts_tablet_manager.cc:1428] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:04:50.733569 2631 raft_consensus.cc:482] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:04:50.733677 2631 raft_consensus.cc:504] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.734014 2631 leader_election.cc:288] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401), 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3:34471)
I20260926 01:04:50.738233 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.734439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43571 (local address 127.0.98.3:34471)
0926 01:04:50.734493 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.734495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.734569 (+ 74us) server_negotiation.cc:407] Connection header received
0926 01:04:50.734627 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.734627 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.734636 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.734708 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:50.734837 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.735607 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.737179 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.737756 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.737843 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.737859 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.738111 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.738163 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:04:50.738233 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.734538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53999 (local address 127.0.98.2:44401)
0926 01:04:50.734594 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.734596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.734665 (+ 69us) server_negotiation.cc:407] Connection header received
0926 01:04:50.734686 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.734687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.734695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.734748 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:04:50.734855 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.735604 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.737198 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.737792 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.737851 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.737859 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.738110 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.738160 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:04:50.738229 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.734324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34471 (local address 127.0.98.1:43571)
0926 01:04:50.734489 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.734500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.734569 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.734715 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.734715 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.734717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:50.734779 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.734781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.735740 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.735741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.737070 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.737073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.737897 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.737898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.737983 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.737990 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.738058 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.738109 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":73,"thread_start_us":105,"threads_started":1}
I20260926 01:04:50.738229 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.734324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44401 (local address 127.0.98.1:53999)
0926 01:04:50.734489 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.734500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.734569 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.734759 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.734760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.734762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:04:50.734796 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.734798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.735716 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.735718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.737070 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.737073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.737902 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.737903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.737983 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.737988 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.738058 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.738109 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":40,"thread_start_us":92,"threads_started":1}
I20260926 01:04:50.738564 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3040604cdcca4e46a9e504a1d29aeb5a" candidate_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b76215cc14f457eb2c1497d54b02e0c" is_pre_election: true
I20260926 01:04:50.738621 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3040604cdcca4e46a9e504a1d29aeb5a" candidate_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" is_pre_election: true
I20260926 01:04:50.738703 2417 raft_consensus.cc:2454] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9296e0d3ec3a4d01ac0271a2ceaaa66d in term 0.
I20260926 01:04:50.738721 2558 raft_consensus.cc:2454] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9296e0d3ec3a4d01ac0271a2ceaaa66d in term 0.
I20260926 01:04:50.738924 2212 leader_election.cc:302] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b76215cc14f457eb2c1497d54b02e0c, 9296e0d3ec3a4d01ac0271a2ceaaa66d; no voters:
I20260926 01:04:50.739063 2631 raft_consensus.cc:2781] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:04:50.739120 2631 raft_consensus.cc:482] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:04:50.739145 2631 raft_consensus.cc:3037] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:50.739940 2631 raft_consensus.cc:504] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.740074 2631 leader_election.cc:288] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [CANDIDATE]: Term 1 election: Requested vote from peers 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401), 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3:34471)
I20260926 01:04:50.740259 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3040604cdcca4e46a9e504a1d29aeb5a" candidate_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b76215cc14f457eb2c1497d54b02e0c"
I20260926 01:04:50.740311 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3040604cdcca4e46a9e504a1d29aeb5a" candidate_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73d76125374f7ebdb69916c2bb2ad4"
I20260926 01:04:50.740351 2558 raft_consensus.cc:3037] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:50.740374 2417 raft_consensus.cc:3037] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:50.741185 2558 raft_consensus.cc:2454] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9296e0d3ec3a4d01ac0271a2ceaaa66d in term 1.
I20260926 01:04:50.741185 2417 raft_consensus.cc:2454] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9296e0d3ec3a4d01ac0271a2ceaaa66d in term 1.
I20260926 01:04:50.741377 2212 leader_election.cc:302] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b76215cc14f457eb2c1497d54b02e0c, 9296e0d3ec3a4d01ac0271a2ceaaa66d; no voters:
I20260926 01:04:50.741472 2631 raft_consensus.cc:2781] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:50.741623 2631 raft_consensus.cc:686] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 LEADER]: Becoming Leader. State: Replica: 9296e0d3ec3a4d01ac0271a2ceaaa66d, State: Running, Role: LEADER
I20260926 01:04:50.741703 2631 consensus_queue.cc:237] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:50.742509 2121 catalog_manager.cc:5495] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } health_report { overall_health: UNKNOWN } } }
I20260926 01:04:50.792958 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.790423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.0.98.1:38025)
0926 01:04:50.790537 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.790539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.790547 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:04:50.790582 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.790583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.790602 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.790675 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:50.790876 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.791454 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.792248 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.792447 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.792502 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.792674 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:50.792788 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:50.792843 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.792872 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.792890 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260926 01:04:50.797775 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.795679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55885 (local address 127.0.98.3:34471)
0926 01:04:50.795718 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.795720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.795830 (+ 110us) server_negotiation.cc:407] Connection header received
0926 01:04:50.795875 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.795876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.795884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.795929 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:50.796060 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.796505 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.797214 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.797390 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.797434 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.797606 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:50.797630 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:50.797661 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.797700 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.797714 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:04:50.797804 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.795615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34471 (local address 127.0.98.1:55885)
0926 01:04:50.795772 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.795783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.795834 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.795937 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.795938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.795939 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:04:50.795990 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.795991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.796570 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.796571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.797155 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.797157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.797456 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.797457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.797535 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.797549 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:04:50.797668 (+ 119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:04:50.797670 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.797697 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.797735 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260926 01:04:50.800381 2558 raft_consensus.cc:1261] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Refusing update from remote peer 9296e0d3ec3a4d01ac0271a2ceaaa66d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:50.800374 2417 raft_consensus.cc:1261] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Refusing update from remote peer 9296e0d3ec3a4d01ac0271a2ceaaa66d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:50.800637 2672 consensus_queue.cc:1035] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:50.800774 2631 consensus_queue.cc:1035] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:50.807587 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.805545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58789 (local address 127.0.98.1:38025)
0926 01:04:50.805581 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:50.805583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:50.805591 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:04:50.805645 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:50.805645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:50.805654 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:50.805693 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:50.805814 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.806284 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.806978 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.807184 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.807229 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.807358 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:50.807393 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:50.807440 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:50.807483 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 01:04:50.807507 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 01:04:50.807606 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:50.805500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38025 (local address 127.0.98.3:58789)
0926 01:04:50.805543 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:50.805556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:50.805595 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:50.805698 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:50.805699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:50.805701 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:04:50.805750 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.805752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.806351 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.806352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.806912 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:50.806913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:50.807233 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:50.807234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:50.807290 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:50.807295 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:04:50.807438 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:04:50.807439 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:04:50.807481 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 01:04:50.807526 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 01:04:50.807965 2650 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 01:04:52.328015 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:52.328130 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:52.328933 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24: Sending DeleteTablet for 3 replicas of tablet 3040604cdcca4e46a9e504a1d29aeb5a
I20260926 01:04:52.329182 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 3040604cdcca4e46a9e504a1d29aeb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50836
I20260926 01:04:52.329218 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 3040604cdcca4e46a9e504a1d29aeb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47682
I20260926 01:04:52.329242 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 3040604cdcca4e46a9e504a1d29aeb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35436
I20260926 01:04:52.329418 2705 tablet_replica.cc:330] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c: stopping tablet replica
I20260926 01:04:52.329473 2705 raft_consensus.cc:2229] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:52.329511 2705 raft_consensus.cc:2258] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:52.330886 2705 ts_tablet_manager.cc:1905] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:52.330977 2706 tablet_replica.cc:330] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d: stopping tablet replica
I20260926 01:04:52.331029 2706 raft_consensus.cc:2229] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:04:52.331125 2706 raft_consensus.cc:2258] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:52.331365 2706 ts_tablet_manager.cc:1905] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:52.333634 2704 tablet_replica.cc:330] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4: stopping tablet replica
I20260926 01:04:52.333702 2704 raft_consensus.cc:2229] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:52.333738 2704 raft_consensus.cc:2258] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:52.333956 2704 ts_tablet_manager.cc:1905] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 01:04:52 dist-test-slave-l73v krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384692, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 01:04:52.351490 2704 ts_tablet_manager.cc:1918] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 01:04:52.351578 2704 log.cc:1196] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/3040604cdcca4e46a9e504a1d29aeb5a
I20260926 01:04:52.351863 2704 ts_tablet_manager.cc:1939] T 3040604cdcca4e46a9e504a1d29aeb5a P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting consensus metadata
I20260926 01:04:52.352077 2705 ts_tablet_manager.cc:1918] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 01:04:52.352149 2705 log.cc:1196] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/3040604cdcca4e46a9e504a1d29aeb5a
I20260926 01:04:52.352382 2705 ts_tablet_manager.cc:1939] T 3040604cdcca4e46a9e504a1d29aeb5a P 3b76215cc14f457eb2c1497d54b02e0c: Deleting consensus metadata
I20260926 01:04:52.355857 2706 ts_tablet_manager.cc:1918] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 01:04:52.355939 2706 log.cc:1196] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/3040604cdcca4e46a9e504a1d29aeb5a
I20260926 01:04:52.356132 2706 ts_tablet_manager.cc:1939] T 3040604cdcca4e46a9e504a1d29aeb5a P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Deleting consensus metadata
Sep 26 01:04:52 dist-test-slave-l73v krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384692, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:52.364058 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:52.352641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.0.98.62:39833)
0926 01:04:52.352904 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:52.352906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:52.352914 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:04:52.352939 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:52.352940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:52.352950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:52.353011 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:52.353613 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:52.354157 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:52.354952 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:52.355161 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:52.355226 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:52.361205 (+ 5979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:52.361207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:52.361207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:52.361215 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:52.362719 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:52.363097 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:52.363099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:52.363099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:52.363171 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:52.363438 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:52.363439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:52.363439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:52.363569 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:52.363686 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:52.363868 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 01:04:52.363948 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":88,"threads_started":1}
I20260926 01:04:52.365742 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:04:52.365904 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:04:52.367599 2398 tablet_service.cc:1467] Processing CreateTablet for tablet b81c387c3f3048e787a90b73aac2eb70 (DEFAULT_TABLE table=test-table [id=468a3f672bc24e1bb76b77a4e9c1ab37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:52.367612 2258 tablet_service.cc:1467] Processing CreateTablet for tablet b81c387c3f3048e787a90b73aac2eb70 (DEFAULT_TABLE table=test-table [id=468a3f672bc24e1bb76b77a4e9c1ab37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:52.367626 2538 tablet_service.cc:1467] Processing CreateTablet for tablet b81c387c3f3048e787a90b73aac2eb70 (DEFAULT_TABLE table=test-table [id=468a3f672bc24e1bb76b77a4e9c1ab37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:52.367758 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81c387c3f3048e787a90b73aac2eb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:52.367761 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81c387c3f3048e787a90b73aac2eb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:52.367803 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81c387c3f3048e787a90b73aac2eb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:52.368942 2715 tablet_bootstrap.cc:492] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Bootstrap starting.
I20260926 01:04:52.369306 2716 tablet_bootstrap.cc:492] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c: Bootstrap starting.
I20260926 01:04:52.369334 2715 tablet_bootstrap.cc:654] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:52.369640 2717 tablet_bootstrap.cc:492] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4: Bootstrap starting.
I20260926 01:04:52.369661 2716 tablet_bootstrap.cc:654] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:52.369884 2715 tablet_bootstrap.cc:492] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: No bootstrap required, opened a new log
I20260926 01:04:52.369943 2715 ts_tablet_manager.cc:1397] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:52.370018 2717 tablet_bootstrap.cc:654] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:52.370065 2715 raft_consensus.cc:348] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.370150 2715 raft_consensus.cc:374] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:52.370175 2715 raft_consensus.cc:729] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9296e0d3ec3a4d01ac0271a2ceaaa66d, State: Initialized, Role: FOLLOWER
I20260926 01:04:52.370222 2715 consensus_queue.cc:260] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.370287 2716 tablet_bootstrap.cc:492] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c: No bootstrap required, opened a new log
I20260926 01:04:52.370311 2715 ts_tablet_manager.cc:1428] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:52.370335 2716 ts_tablet_manager.cc:1397] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:52.370808 2717 tablet_bootstrap.cc:492] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4: No bootstrap required, opened a new log
I20260926 01:04:52.370859 2717 ts_tablet_manager.cc:1397] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:52.371023 2716 raft_consensus.cc:348] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.371104 2716 raft_consensus.cc:374] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:52.371124 2716 raft_consensus.cc:729] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b76215cc14f457eb2c1497d54b02e0c, State: Initialized, Role: FOLLOWER
I20260926 01:04:52.371186 2716 consensus_queue.cc:260] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.371198 2717 raft_consensus.cc:348] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.371290 2717 raft_consensus.cc:374] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:52.371289 2716 ts_tablet_manager.cc:1428] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:52.371309 2717 raft_consensus.cc:729] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e73d76125374f7ebdb69916c2bb2ad4, State: Initialized, Role: FOLLOWER
I20260926 01:04:52.371348 2717 consensus_queue.cc:260] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.371513 2717 ts_tablet_manager.cc:1428] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:52.372331 2106 catalog_manager.cc:4841] TS 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401): tablet 3040604cdcca4e46a9e504a1d29aeb5a (table test-table [id=5cccf6492acf4fea8cbfd97820d7b648]) successfully deleted
I20260926 01:04:52.373651 2108 catalog_manager.cc:4841] TS 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025): tablet 3040604cdcca4e46a9e504a1d29aeb5a (table test-table [id=5cccf6492acf4fea8cbfd97820d7b648]) successfully deleted
I20260926 01:04:52.373740 2108 catalog_manager.cc:4841] TS 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3:34471): tablet 3040604cdcca4e46a9e504a1d29aeb5a (table test-table [id=5cccf6492acf4fea8cbfd97820d7b648]) successfully deleted
I20260926 01:04:52.614302 2689 raft_consensus.cc:482] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:04:52.614439 2689 raft_consensus.cc:504] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.614612 2689 leader_election.cc:288] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025), 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401)
I20260926 01:04:52.614944 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81c387c3f3048e787a90b73aac2eb70" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" is_pre_election: true
I20260926 01:04:52.614928 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81c387c3f3048e787a90b73aac2eb70" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" is_pre_election: true
I20260926 01:04:52.615041 2417 raft_consensus.cc:2454] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 0.
I20260926 01:04:52.615041 2278 raft_consensus.cc:2454] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 0.
I20260926 01:04:52.615217 2490 leader_election.cc:302] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b76215cc14f457eb2c1497d54b02e0c, 9e73d76125374f7ebdb69916c2bb2ad4; no voters:
I20260926 01:04:52.615314 2689 raft_consensus.cc:2781] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:04:52.615353 2689 raft_consensus.cc:482] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:04:52.615371 2689 raft_consensus.cc:3037] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:52.616084 2689 raft_consensus.cc:504] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.616220 2689 leader_election.cc:288] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 election: Requested vote from peers 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025), 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401)
I20260926 01:04:52.616407 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81c387c3f3048e787a90b73aac2eb70" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d"
I20260926 01:04:52.616439 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81c387c3f3048e787a90b73aac2eb70" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73d76125374f7ebdb69916c2bb2ad4"
I20260926 01:04:52.616490 2278 raft_consensus.cc:3037] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:52.616503 2417 raft_consensus.cc:3037] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:52.617115 2417 raft_consensus.cc:2454] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 1.
I20260926 01:04:52.617187 2278 raft_consensus.cc:2454] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 1.
I20260926 01:04:52.617291 2490 leader_election.cc:302] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b76215cc14f457eb2c1497d54b02e0c, 9e73d76125374f7ebdb69916c2bb2ad4; no voters:
I20260926 01:04:52.617380 2689 raft_consensus.cc:2781] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:52.617439 2689 raft_consensus.cc:686] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 1 LEADER]: Becoming Leader. State: Replica: 3b76215cc14f457eb2c1497d54b02e0c, State: Running, Role: LEADER
I20260926 01:04:52.617499 2689 consensus_queue.cc:237] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.618125 2121 catalog_manager.cc:5495] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } health_report { overall_health: HEALTHY } } }
I20260926 01:04:52.658829 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:52.656528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.0.98.3:34471)
0926 01:04:52.656690 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:52.656692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:52.656698 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:04:52.656723 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:52.656724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:52.656735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:52.656789 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:04:52.656925 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:52.657421 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:52.658152 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:52.658357 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:52.658448 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:52.658574 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:04:52.658609 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:04:52.658653 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:52.658720 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 01:04:52.658742 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260926 01:04:52.661053 2417 raft_consensus.cc:1261] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Refusing update from remote peer 3b76215cc14f457eb2c1497d54b02e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:52.661087 2278 raft_consensus.cc:1261] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Refusing update from remote peer 3b76215cc14f457eb2c1497d54b02e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:52.661289 2684 consensus_queue.cc:1035] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:52.661409 2689 consensus_queue.cc:1035] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:52.671375 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:52.671491 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:52.672185 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24: Sending DeleteTablet for 3 replicas of tablet b81c387c3f3048e787a90b73aac2eb70
I20260926 01:04:52.672524 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet b81c387c3f3048e787a90b73aac2eb70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35436
I20260926 01:04:52.672529 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet b81c387c3f3048e787a90b73aac2eb70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47682
I20260926 01:04:52.672523 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet b81c387c3f3048e787a90b73aac2eb70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50836
I20260926 01:04:52.672623 2706 tablet_replica.cc:330] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: stopping tablet replica
I20260926 01:04:52.672626 2704 tablet_replica.cc:330] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4: stopping tablet replica
I20260926 01:04:52.672672 2704 raft_consensus.cc:2229] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:52.672676 2706 raft_consensus.cc:2229] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:52.672710 2704 raft_consensus.cc:2258] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:52.672710 2706 raft_consensus.cc:2258] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:52.672832 2705 tablet_replica.cc:330] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c: stopping tablet replica
I20260926 01:04:52.672889 2705 raft_consensus.cc:2229] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:04:52.672871 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:04:52.672945 2705 raft_consensus.cc:2258] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:04:52.673002 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:04:52.673322 2705 ts_tablet_manager.cc:1905] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:52.673463 2706 ts_tablet_manager.cc:1905] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:52.675231 2705 ts_tablet_manager.cc:1918] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:04:52.675241 2704 ts_tablet_manager.cc:1905] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:52.675299 2705 log.cc:1196] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/b81c387c3f3048e787a90b73aac2eb70
I20260926 01:04:52.675510 2705 ts_tablet_manager.cc:1939] T b81c387c3f3048e787a90b73aac2eb70 P 3b76215cc14f457eb2c1497d54b02e0c: Deleting consensus metadata
I20260926 01:04:52.675895 2398 tablet_service.cc:1467] Processing CreateTablet for tablet f7f7ba6cc715451595bfb65d5c01e5dc (DEFAULT_TABLE table=test-table [id=ed0f38f58913471282a1ceeb4278127f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:52.675915 2538 tablet_service.cc:1467] Processing CreateTablet for tablet f7f7ba6cc715451595bfb65d5c01e5dc (DEFAULT_TABLE table=test-table [id=ed0f38f58913471282a1ceeb4278127f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:52.676007 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f7ba6cc715451595bfb65d5c01e5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:52.676007 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f7ba6cc715451595bfb65d5c01e5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:52.676676 2258 tablet_service.cc:1467] Processing CreateTablet for tablet f7f7ba6cc715451595bfb65d5c01e5dc (DEFAULT_TABLE table=test-table [id=ed0f38f58913471282a1ceeb4278127f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:04:52.676718 2704 ts_tablet_manager.cc:1918] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:04:52.676769 2704 log.cc:1196] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/b81c387c3f3048e787a90b73aac2eb70
I20260926 01:04:52.676823 2706 ts_tablet_manager.cc:1918] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:04:52.676863 2706 log.cc:1196] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/b81c387c3f3048e787a90b73aac2eb70
I20260926 01:04:52.677069 2704 ts_tablet_manager.cc:1939] T b81c387c3f3048e787a90b73aac2eb70 P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting consensus metadata
I20260926 01:04:52.677078 2706 ts_tablet_manager.cc:1939] T b81c387c3f3048e787a90b73aac2eb70 P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Deleting consensus metadata
I20260926 01:04:52.677202 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f7ba6cc715451595bfb65d5c01e5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:52.677273 2716 tablet_bootstrap.cc:492] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c: Bootstrap starting.
I20260926 01:04:52.677492 2108 catalog_manager.cc:4841] TS 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3:34471): tablet b81c387c3f3048e787a90b73aac2eb70 (table test-table [id=468a3f672bc24e1bb76b77a4e9c1ab37]) successfully deleted
I20260926 01:04:52.677589 2716 tablet_bootstrap.cc:654] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:52.678092 2716 tablet_bootstrap.cc:492] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c: No bootstrap required, opened a new log
I20260926 01:04:52.678134 2716 ts_tablet_manager.cc:1397] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:52.678242 2716 raft_consensus.cc:348] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.678306 2716 raft_consensus.cc:374] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:52.678318 2716 raft_consensus.cc:729] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b76215cc14f457eb2c1497d54b02e0c, State: Initialized, Role: FOLLOWER
I20260926 01:04:52.678360 2716 consensus_queue.cc:260] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.678424 2715 tablet_bootstrap.cc:492] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Bootstrap starting.
I20260926 01:04:52.678431 2717 tablet_bootstrap.cc:492] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4: Bootstrap starting.
I20260926 01:04:52.678479 2716 ts_tablet_manager.cc:1428] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:52.678750 2715 tablet_bootstrap.cc:654] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:52.678822 2717 tablet_bootstrap.cc:654] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:52.679307 2715 tablet_bootstrap.cc:492] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d: No bootstrap required, opened a new log
I20260926 01:04:52.679349 2715 ts_tablet_manager.cc:1397] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:52.679361 2717 tablet_bootstrap.cc:492] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4: No bootstrap required, opened a new log
I20260926 01:04:52.679385 2106 catalog_manager.cc:4841] TS 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401): tablet b81c387c3f3048e787a90b73aac2eb70 (table test-table [id=468a3f672bc24e1bb76b77a4e9c1ab37]) successfully deleted
I20260926 01:04:52.679401 2717 ts_tablet_manager.cc:1397] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:52.679455 2715 raft_consensus.cc:348] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.679517 2715 raft_consensus.cc:374] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:52.679533 2715 raft_consensus.cc:729] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9296e0d3ec3a4d01ac0271a2ceaaa66d, State: Initialized, Role: FOLLOWER
I20260926 01:04:52.679538 2717 raft_consensus.cc:348] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.679577 2715 consensus_queue.cc:260] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.679605 2717 raft_consensus.cc:374] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:52.679625 2717 raft_consensus.cc:729] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e73d76125374f7ebdb69916c2bb2ad4, State: Initialized, Role: FOLLOWER
I20260926 01:04:52.679657 2715 ts_tablet_manager.cc:1428] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:52.679657 2717 consensus_queue.cc:260] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.679736 2717 ts_tablet_manager.cc:1428] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:52.680018 2108 catalog_manager.cc:4841] TS 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025): tablet b81c387c3f3048e787a90b73aac2eb70 (table test-table [id=468a3f672bc24e1bb76b77a4e9c1ab37]) successfully deleted
I20260926 01:04:52.831359 2634 raft_consensus.cc:482] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:04:52.831465 2634 raft_consensus.cc:504] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.831629 2634 leader_election.cc:288] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025), 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401)
I20260926 01:04:52.831905 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f7ba6cc715451595bfb65d5c01e5dc" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" is_pre_election: true
I20260926 01:04:52.831916 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f7ba6cc715451595bfb65d5c01e5dc" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" is_pre_election: true
I20260926 01:04:52.831996 2278 raft_consensus.cc:2454] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 0.
I20260926 01:04:52.832011 2417 raft_consensus.cc:2454] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 0.
I20260926 01:04:52.832199 2490 leader_election.cc:302] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b76215cc14f457eb2c1497d54b02e0c, 9e73d76125374f7ebdb69916c2bb2ad4; no voters:
I20260926 01:04:52.832306 2634 raft_consensus.cc:2781] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:04:52.832346 2634 raft_consensus.cc:482] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:04:52.832366 2634 raft_consensus.cc:3037] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:52.833060 2634 raft_consensus.cc:504] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.833194 2634 leader_election.cc:288] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 election: Requested vote from peers 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025), 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401)
I20260926 01:04:52.833374 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f7ba6cc715451595bfb65d5c01e5dc" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d"
I20260926 01:04:52.833371 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f7ba6cc715451595bfb65d5c01e5dc" candidate_uuid: "3b76215cc14f457eb2c1497d54b02e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e73d76125374f7ebdb69916c2bb2ad4"
I20260926 01:04:52.833461 2417 raft_consensus.cc:3037] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:52.833462 2278 raft_consensus.cc:3037] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:52.834043 2278 raft_consensus.cc:2454] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 1.
I20260926 01:04:52.834092 2417 raft_consensus.cc:2454] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b76215cc14f457eb2c1497d54b02e0c in term 1.
I20260926 01:04:52.834255 2490 leader_election.cc:302] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b76215cc14f457eb2c1497d54b02e0c, 9296e0d3ec3a4d01ac0271a2ceaaa66d, 9e73d76125374f7ebdb69916c2bb2ad4; no voters:
I20260926 01:04:52.834410 2634 raft_consensus.cc:2781] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:52.834486 2634 raft_consensus.cc:686] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 1 LEADER]: Becoming Leader. State: Replica: 3b76215cc14f457eb2c1497d54b02e0c, State: Running, Role: LEADER
I20260926 01:04:52.834537 2634 consensus_queue.cc:237] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } }
I20260926 01:04:52.835141 2119 catalog_manager.cc:5495] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3b76215cc14f457eb2c1497d54b02e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b76215cc14f457eb2c1497d54b02e0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34471 } health_report { overall_health: HEALTHY } } }
I20260926 01:04:52.866321 2417 raft_consensus.cc:1261] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Refusing update from remote peer 3b76215cc14f457eb2c1497d54b02e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:52.866423 2278 raft_consensus.cc:1261] T f7f7ba6cc715451595bfb65d5c01e5dc P 9296e0d3ec3a4d01ac0271a2ceaaa66d [term 1 FOLLOWER]: Refusing update from remote peer 3b76215cc14f457eb2c1497d54b02e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:04:52.866595 2689 consensus_queue.cc:1035] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e73d76125374f7ebdb69916c2bb2ad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:04:52.866706 2684 consensus_queue.cc:1035] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9296e0d3ec3a4d01ac0271a2ceaaa66d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:04:52.871358 2682 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 01:04:54.388813 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:54.388913 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:04:54.389775 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 630a8784127349ff9f7cc66d13d6ec24: Sending DeleteTablet for 3 replicas of tablet f7f7ba6cc715451595bfb65d5c01e5dc
I20260926 01:04:54.390038 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f7ba6cc715451595bfb65d5c01e5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50836
I20260926 01:04:54.390070 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f7ba6cc715451595bfb65d5c01e5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35436
I20260926 01:04:54.390151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 2193
I20260926 01:04:54.391989 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f7ba6cc715451595bfb65d5c01e5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:04:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47682
I20260926 01:04:54.392203 2755 tablet_replica.cc:330] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c: stopping tablet replica
I20260926 01:04:54.392228 2753 tablet_replica.cc:330] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4: stopping tablet replica
I20260926 01:04:54.392261 2755 raft_consensus.cc:2229] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:04:54.392285 2753 raft_consensus.cc:2229] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:04:54.392320 2753 raft_consensus.cc:2258] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:54.392355 2755 raft_consensus.cc:2258] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:04:54.392593 2755 ts_tablet_manager.cc:1905] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:54.392642 2753 ts_tablet_manager.cc:1905] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:04:54.394230 2755 ts_tablet_manager.cc:1918] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 01:04:54.394286 2755 log.cc:1196] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/f7f7ba6cc715451595bfb65d5c01e5dc
I20260926 01:04:54.394273 2753 ts_tablet_manager.cc:1918] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 01:04:54.394320 2753 log.cc:1196] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/f7f7ba6cc715451595bfb65d5c01e5dc
I20260926 01:04:54.394534 2755 ts_tablet_manager.cc:1939] T f7f7ba6cc715451595bfb65d5c01e5dc P 3b76215cc14f457eb2c1497d54b02e0c: Deleting consensus metadata
I20260926 01:04:54.394567 2753 ts_tablet_manager.cc:1939] T f7f7ba6cc715451595bfb65d5c01e5dc P 9e73d76125374f7ebdb69916c2bb2ad4: Deleting consensus metadata
I20260926 01:04:54.394917 2108 catalog_manager.cc:4841] TS 3b76215cc14f457eb2c1497d54b02e0c (127.0.98.3:34471): tablet f7f7ba6cc715451595bfb65d5c01e5dc (table test-table [id=ed0f38f58913471282a1ceeb4278127f]) successfully deleted
I20260926 01:04:54.394951 2106 catalog_manager.cc:4841] TS 9e73d76125374f7ebdb69916c2bb2ad4 (127.0.98.2:44401): tablet f7f7ba6cc715451595bfb65d5c01e5dc (table test-table [id=ed0f38f58913471282a1ceeb4278127f]) successfully deleted
W20260926 01:04:54.396129 2108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38025 (error 108)
I20260926 01:04:54.396333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 2333
I20260926 01:04:54.397178 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:54.396495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38025 (local address 127.0.0.1:54744)
0926 01:04:54.397087 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:54.397137 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":59,"threads_started":1}
W20260926 01:04:54.397267 2108 catalog_manager.cc:4584] TS 9296e0d3ec3a4d01ac0271a2ceaaa66d (127.0.98.1:38025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7f7ba6cc715451595bfb65d5c01e5dc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38025: connect: Connection refused (error 111)
I20260926 01:04:54.401003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 2473
I20260926 01:04:54.406257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 2089
2026-09-26T01:04:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6099 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:04:54 dist-test-slave-l73v krb5kdc[2760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:04:54 dist-test-slave-l73v krb5kdc[2760](info): set up 2 sockets
Sep 26 01:04:54 dist-test-slave-l73v krb5kdc[2760](info): commencing operation
krb5kdc: starting...
W20260926 01:04:55.378192 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:04:55 dist-test-slave-l73v krb5kdc[2760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:04:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:04:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:04:55.499586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37163
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:55.585055 2773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:55.585364 2773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:55.585404 2773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:55.587172 2773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:04:55.587239 2773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:55.587260 2773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:55.587276 2773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:04:55.587330 2773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:04:55.589730 2773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:55.590062 2773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:55.590348 2773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:55.595707 2779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:55.595707 2773 server_base.cc:1034] running on GCE node
W20260926 01:04:55.595819 2778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:55.595988 2781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:55.596282 2773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:55.596650 2773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:55.597795 2773 hybrid_clock.cc:648] HybridClock initialized: now 1790384695597780 us; error 31 us; skew 500 ppm
Sep 26 01:04:55 dist-test-slave-l73v krb5kdc[2760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:55.601126 2773 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:04:55.601578 2773 webserver.cc:460] Webserver started at http://127.0.98.62:37533/ using document root <none> and password file <none>
I20260926 01:04:55.601792 2773 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:55.601840 2773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:55.601966 2773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:55.603261 2773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "5504cc6c93a3435db2e70370a07ea4b4"
format_stamp: "Formatted at 2026-09-26 01:04:55 on dist-test-slave-l73v"
server_key: "0b787caf8f19684ea4dad96f9c9519d7"
server_key_iv: "7a413ce88f94f4d8c405542306da3f08"
server_key_version: "encryptionkey@0"
I20260926 01:04:55.603657 2773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "5504cc6c93a3435db2e70370a07ea4b4"
format_stamp: "Formatted at 2026-09-26 01:04:55 on dist-test-slave-l73v"
server_key: "0b787caf8f19684ea4dad96f9c9519d7"
server_key_iv: "7a413ce88f94f4d8c405542306da3f08"
server_key_version: "encryptionkey@0"
I20260926 01:04:55.605369 2773 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 01:04:55.606299 2788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:55.606523 2773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:55.606631 2773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "5504cc6c93a3435db2e70370a07ea4b4"
format_stamp: "Formatted at 2026-09-26 01:04:55 on dist-test-slave-l73v"
server_key: "0b787caf8f19684ea4dad96f9c9519d7"
server_key_iv: "7a413ce88f94f4d8c405542306da3f08"
server_key_version: "encryptionkey@0"
I20260926 01:04:55.606714 2773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:55.620926 2773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:55.623293 2773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:55.623446 2773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:55.627281 2773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37163
I20260926 01:04:55.627327 2840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37163 every 8 connection(s)
I20260926 01:04:55.627810 2773 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:04:55.628504 2841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:55.630847 2841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4: Bootstrap starting.
I20260926 01:04:55.631423 2841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:55.631685 2841 log.cc:824] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:55.632378 2841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4: No bootstrap required, opened a new log
I20260926 01:04:55.634007 2841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5504cc6c93a3435db2e70370a07ea4b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } }
I20260926 01:04:55.634166 2841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:55.634189 2841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5504cc6c93a3435db2e70370a07ea4b4, State: Initialized, Role: FOLLOWER
I20260926 01:04:55.634325 2841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5504cc6c93a3435db2e70370a07ea4b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } }
I20260926 01:04:55.634420 2841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:04:55.634451 2841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:04:55.634491 2841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:55.634558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 2773
I20260926 01:04:55.634619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:04:55.634763 392 external_mini_cluster.cc:1428] Setting key 21525685a53342648ef0f345b6bf33fd
I20260926 01:04:55.635522 2841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5504cc6c93a3435db2e70370a07ea4b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } }
I20260926 01:04:55.635623 2841 leader_election.cc:302] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5504cc6c93a3435db2e70370a07ea4b4; no voters:
I20260926 01:04:55.635775 2841 leader_election.cc:288] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:04:55.635816 2846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:55.635914 2846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [term 1 LEADER]: Becoming Leader. State: Replica: 5504cc6c93a3435db2e70370a07ea4b4, State: Running, Role: LEADER
I20260926 01:04:55.635995 2846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5504cc6c93a3435db2e70370a07ea4b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } }
I20260926 01:04:55.636035 2841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:04:55.636333 2847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5504cc6c93a3435db2e70370a07ea4b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5504cc6c93a3435db2e70370a07ea4b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } } }
I20260926 01:04:55.636399 2847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:55.636528 2848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5504cc6c93a3435db2e70370a07ea4b4. Latest consensus state: current_term: 1 leader_uuid: "5504cc6c93a3435db2e70370a07ea4b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5504cc6c93a3435db2e70370a07ea4b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } } }
I20260926 01:04:55.636607 2848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:55.636724 2851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:04:55.637225 2851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:04:55.638798 2851 catalog_manager.cc:1347] Generated new cluster ID: 28a2530f28ce4aae8e646d0e5a17f31a
I20260926 01:04:55.638841 2851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:04:55 dist-test-slave-l73v krb5kdc[2760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:55.646762 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:55.635264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.98.62:37163)
0926 01:04:55.635431 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:55.635439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:55.635465 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:04:55.635571 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:55.635577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:55.635645 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:55.635778 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:55.636487 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:55.637865 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:55.638955 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:55.639553 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:55.639679 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:55.642485 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:55.642489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:55.642490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:55.642502 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:55.644979 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:55.645387 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:55.645390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:55.645390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:55.645495 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:55.645698 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:55.645699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:55.645699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:55.645987 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:55.646133 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:55.646250 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 01:04:55.646514 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260926 01:04:55.660490 2851 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:04:55.661278 2851 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:04:55.669694 2851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5504cc6c93a3435db2e70370a07ea4b4: Generated new TSK 0
I20260926 01:04:55.669934 2851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:04:55.698022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:55.783741 2867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:55.783967 2867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:55.783993 2867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:55.785218 2867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:55.785266 2867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:55.785293 2867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:04:55.786916 2867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:55.787182 2867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:55.787420 2867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:55.792474 2872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:55.792502 2875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:55.792558 2873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:55.792636 2867 server_base.cc:1034] running on GCE node
I20260926 01:04:55.792949 2867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:55.793206 2867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:55.794407 2867 hybrid_clock.cc:648] HybridClock initialized: now 1790384695794363 us; error 28 us; skew 500 ppm
Sep 26 01:04:55 dist-test-slave-l73v krb5kdc[2760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:55.797125 2867 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:04:55.797618 2867 webserver.cc:460] Webserver started at http://127.0.98.1:34855/ using document root <none> and password file <none>
I20260926 01:04:55.797844 2867 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:55.797895 2867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:55.798024 2867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:55.799177 2867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "b8d5a2e1b9914beba24115484e3b594c"
format_stamp: "Formatted at 2026-09-26 01:04:55 on dist-test-slave-l73v"
server_key: "42cc466068c03005aadd9ef8682c1d6e"
server_key_iv: "f115ad99c424517a1701db36a4e448d8"
server_key_version: "encryptionkey@0"
I20260926 01:04:55.799528 2867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b8d5a2e1b9914beba24115484e3b594c"
format_stamp: "Formatted at 2026-09-26 01:04:55 on dist-test-slave-l73v"
server_key: "42cc466068c03005aadd9ef8682c1d6e"
server_key_iv: "f115ad99c424517a1701db36a4e448d8"
server_key_version: "encryptionkey@0"
I20260926 01:04:55.800920 2867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:04:55.801836 2882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:55.802006 2867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:55.802090 2867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "b8d5a2e1b9914beba24115484e3b594c"
format_stamp: "Formatted at 2026-09-26 01:04:55 on dist-test-slave-l73v"
server_key: "42cc466068c03005aadd9ef8682c1d6e"
server_key_iv: "f115ad99c424517a1701db36a4e448d8"
server_key_version: "encryptionkey@0"
I20260926 01:04:55.802153 2867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:55.815392 2867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:55.817710 2867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:55.817864 2867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:55.818133 2867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:55.818481 2867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:55.818518 2867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:55.818549 2867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:55.818565 2867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:55.824137 2867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37899
I20260926 01:04:55.824182 2995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37899 every 8 connection(s)
I20260926 01:04:55.824505 2867 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:04:55 dist-test-slave-l73v krb5kdc[2760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:55.832906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 2867
I20260926 01:04:55.832988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:04:55.833141 392 external_mini_cluster.cc:1428] Setting key 68e66c4a42ea1a2f80f7b4d242063744
I20260926 01:04:55.834288 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:55.825296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36299 (local address 127.0.98.62:37163)
0926 01:04:55.825381 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:55.825384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:55.825586 (+ 202us) server_negotiation.cc:407] Connection header received
0926 01:04:55.826608 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:55.826609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:55.826623 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:55.826682 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:55.827396 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:55.827903 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:55.829007 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:55.829220 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:55.829287 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:55.831575 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:55.831577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:55.831577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:55.831584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:55.832920 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:55.833367 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:55.833369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:55.833369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:55.833422 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:55.833653 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:55.833654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:55.833655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:55.833762 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:55.833829 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:55.834105 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 01:04:55.834189 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 01:04:55.834494 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:55.825271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37163 (local address 127.0.98.1:36299)
0926 01:04:55.825450 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:55.825488 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:55.826448 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:55.826756 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:55.826759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:55.827044 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:55.827251 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:55.827255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:55.828025 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:55.828026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:55.828903 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:55.828906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:55.829305 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:55.829306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:55.829418 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:55.829961 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:55.829970 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:55.831472 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:55.833041 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:55.833049 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:55.833052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:55.833304 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:55.833489 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:55.833490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:55.833491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:55.833598 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:55.833829 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:55.833835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:55.833986 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 01:04:55.834077 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 01:04:55.834243 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
I20260926 01:04:55.834919 2996 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37163
I20260926 01:04:55.835017 2996 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:55.835192 2996 heartbeater.cc:511] Master 127.0.98.62:37163 requested a full tablet report, sending...
I20260926 01:04:55.835677 2805 ts_manager.cc:194] Registered new tserver with Master: b8d5a2e1b9914beba24115484e3b594c (127.0.98.1:37899)
I20260926 01:04:55.836340 2805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36299
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:04:55.862219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:55.945142 3001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:55.945380 3001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:55.945407 3001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:55.946892 3001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:55.946944 3001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:55.946976 3001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:04:55.948568 3001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:55.948783 3001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:55.948987 3001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:55.954056 3006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:55.954058 3007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:55.954058 3009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:55.954336 3001 server_base.cc:1034] running on GCE node
I20260926 01:04:55.954573 3001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:55.954807 3001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:55.955974 3001 hybrid_clock.cc:648] HybridClock initialized: now 1790384695955959 us; error 29 us; skew 500 ppm
Sep 26 01:04:55 dist-test-slave-l73v krb5kdc[2760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:55.958542 3001 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:04:55.959012 3001 webserver.cc:460] Webserver started at http://127.0.98.2:36465/ using document root <none> and password file <none>
I20260926 01:04:55.959224 3001 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:55.959280 3001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:55.959414 3001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:55.960498 3001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "a9f4c7eb66764a51b650d404fbccbee7"
format_stamp: "Formatted at 2026-09-26 01:04:55 on dist-test-slave-l73v"
server_key: "b730b8165283fb0d8457aa56d052a03a"
server_key_iv: "c034e2ec83744febec332150fe78abc2"
server_key_version: "encryptionkey@0"
I20260926 01:04:55.960829 3001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a9f4c7eb66764a51b650d404fbccbee7"
format_stamp: "Formatted at 2026-09-26 01:04:55 on dist-test-slave-l73v"
server_key: "b730b8165283fb0d8457aa56d052a03a"
server_key_iv: "c034e2ec83744febec332150fe78abc2"
server_key_version: "encryptionkey@0"
I20260926 01:04:55.962128 3001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:04:55.963070 3016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:55.963275 3001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:04:55.963372 3001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "a9f4c7eb66764a51b650d404fbccbee7"
format_stamp: "Formatted at 2026-09-26 01:04:55 on dist-test-slave-l73v"
server_key: "b730b8165283fb0d8457aa56d052a03a"
server_key_iv: "c034e2ec83744febec332150fe78abc2"
server_key_version: "encryptionkey@0"
I20260926 01:04:55.963438 3001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:55.975334 3001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:55.977500 3001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:55.977648 3001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:55.977873 3001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:55.978175 3001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:55.978209 3001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:55.978238 3001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:55.978255 3001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:55.984102 3001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44145
I20260926 01:04:55.984148 3129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44145 every 8 connection(s)
I20260926 01:04:55.984469 3001 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:04:55.987219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 3001
I20260926 01:04:55.987313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:04:55.987473 392 external_mini_cluster.cc:1428] Setting key 9d1a923c78a9d127ae7d807cfa788a10
Sep 26 01:04:55 dist-test-slave-l73v krb5kdc[2760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:04:55.994930 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:55.985233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42231 (local address 127.0.98.62:37163)
0926 01:04:55.985314 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:55.985316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:55.985481 (+ 165us) server_negotiation.cc:407] Connection header received
0926 01:04:55.986472 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:55.986473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:55.986485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:55.986537 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:55.987178 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:55.987937 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:55.989030 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:55.989275 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:55.989337 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:55.991931 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:55.991933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:55.991934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:55.991942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:55.993491 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:55.993976 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:55.993978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:55.993979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:55.994034 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:55.994262 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:55.994263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:55.994263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:55.994408 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:55.994468 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:55.994743 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 01:04:55.994836 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 01:04:55.995115 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:55.985171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37163 (local address 127.0.98.2:42231)
0926 01:04:55.985351 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:55.985397 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:55.986330 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:55.986610 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:55.986613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:55.986886 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:55.987085 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:55.987090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:55.988061 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:55.988063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:55.988938 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:55.988941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:55.989355 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:55.989357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:55.989480 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:55.990071 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:55.990084 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:55.991811 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:55.993608 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:55.993640 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:55.993643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:55.993900 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:55.994093 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:55.994094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:55.994095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:55.994206 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:55.994459 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:55.994466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:55.994622 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 01:04:55.994715 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 01:04:55.994891 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260926 01:04:55.995572 3130 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37163
I20260926 01:04:55.995667 3130 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:55.995852 3130 heartbeater.cc:511] Master 127.0.98.62:37163 requested a full tablet report, sending...
I20260926 01:04:55.996235 2805 ts_manager.cc:194] Registered new tserver with Master: a9f4c7eb66764a51b650d404fbccbee7 (127.0.98.2:44145)
I20260926 01:04:55.996579 2805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:04:56.014051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:56.096789 3135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:56.097018 3135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:56.097043 3135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:56.098300 3135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:56.098376 3135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:56.098450 3135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:04:56.100147 3135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:56.100399 3135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:56.100631 3135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:56.105924 3140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:56.105901 3143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:56.105899 3141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:56.106211 3135 server_base.cc:1034] running on GCE node
I20260926 01:04:56.106539 3135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:56.106873 3135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:56.108048 3135 hybrid_clock.cc:648] HybridClock initialized: now 1790384696108035 us; error 28 us; skew 500 ppm
Sep 26 01:04:56 dist-test-slave-l73v krb5kdc[2760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:56.110993 3135 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:04:56.111471 3135 webserver.cc:460] Webserver started at http://127.0.98.3:33543/ using document root <none> and password file <none>
I20260926 01:04:56.111690 3135 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:56.111742 3135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:56.111868 3135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:56.112964 3135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "6e10b9d33a5c4cd8b8e630667bcae558"
format_stamp: "Formatted at 2026-09-26 01:04:56 on dist-test-slave-l73v"
server_key: "a74494b4eafdce5e5bba48b1d66703a1"
server_key_iv: "29392520fedbad3190c1be667afdd80b"
server_key_version: "encryptionkey@0"
I20260926 01:04:56.113335 3135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6e10b9d33a5c4cd8b8e630667bcae558"
format_stamp: "Formatted at 2026-09-26 01:04:56 on dist-test-slave-l73v"
server_key: "a74494b4eafdce5e5bba48b1d66703a1"
server_key_iv: "29392520fedbad3190c1be667afdd80b"
server_key_version: "encryptionkey@0"
I20260926 01:04:56.114764 3135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:04:56.115625 3150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:56.115902 3135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:04:56.116017 3135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "6e10b9d33a5c4cd8b8e630667bcae558"
format_stamp: "Formatted at 2026-09-26 01:04:56 on dist-test-slave-l73v"
server_key: "a74494b4eafdce5e5bba48b1d66703a1"
server_key_iv: "29392520fedbad3190c1be667afdd80b"
server_key_version: "encryptionkey@0"
I20260926 01:04:56.116093 3135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:56.142493 3135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:56.144699 3135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:56.144847 3135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:56.145090 3135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:56.145388 3135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:56.145421 3135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:56.145452 3135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:56.145471 3135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:56.150986 3135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42113
I20260926 01:04:56.151046 3263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42113 every 8 connection(s)
I20260926 01:04:56.151340 3135 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:04:56 dist-test-slave-l73v krb5kdc[2760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:56.159457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 3135
I20260926 01:04:56.159533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:04:56.159687 392 external_mini_cluster.cc:1428] Setting key 8d6ebe9ec0d7e4747190629bfc4d298b
I20260926 01:04:56.161233 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:56.152079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46225 (local address 127.0.98.62:37163)
0926 01:04:56.152136 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:56.152138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:56.152304 (+ 166us) server_negotiation.cc:407] Connection header received
0926 01:04:56.153291 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:56.153293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:56.153306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:56.153367 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:56.154024 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:56.154579 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:56.155634 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:56.155833 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:56.155890 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:56.158201 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:56.158203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:56.158203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:56.158210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:56.159740 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:56.160237 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:56.160239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:56.160239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:56.160300 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:56.160577 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:56.160578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:56.160579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:56.160744 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:56.160802 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:56.161083 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 01:04:56.161165 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:04:56.161375 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:56.152050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37163 (local address 127.0.98.3:46225)
0926 01:04:56.152190 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:56.152226 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:56.153179 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:56.153442 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:56.153446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:56.153720 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:56.153935 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:56.153940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:56.154666 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:56.154667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:56.155547 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:56.155550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:56.155909 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:56.155910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:56.156030 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:56.156588 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:56.156597 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:56.158065 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:56.159831 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:56.159842 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:56.159844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:56.160138 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:56.160369 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:56.160370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:56.160370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:56.160510 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:56.160800 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:56.160804 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:56.160936 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 01:04:56.161026 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 01:04:56.161162 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260926 01:04:56.161882 3264 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37163
I20260926 01:04:56.161988 3264 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:56.162184 3264 heartbeater.cc:511] Master 127.0.98.62:37163 requested a full tablet report, sending...
I20260926 01:04:56.162647 2805 ts_manager.cc:194] Registered new tserver with Master: 6e10b9d33a5c4cd8b8e630667bcae558 (127.0.98.3:42113)
I20260926 01:04:56.163028 2805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46225
I20260926 01:04:56.171835 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 01:04:56.182861 2845 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:04:56.181846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.0.98.62:37163)
0926 01:04:56.181931 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:56.181933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:56.182004 (+ 71us) server_negotiation.cc:407] Connection header received
0926 01:04:56.182038 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:56.182039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:56.182049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:56.182105 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:56.182788 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57330: BlockingRecv error: recv got EOF from 127.0.0.1:57330 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 01:04:56.183718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 2867
I20260926 01:04:56.188310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 3001
I20260926 01:04:56.193315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 3135
I20260926 01:04:56.197242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 2773
2026-09-26T01:04:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1790 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:04:56 dist-test-slave-l73v krb5kdc[3276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:04:56 dist-test-slave-l73v krb5kdc[3276](info): set up 2 sockets
Sep 26 01:04:56 dist-test-slave-l73v krb5kdc[3276](info): commencing operation
krb5kdc: starting...
W20260926 01:04:57.166131 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:04:57 dist-test-slave-l73v krb5kdc[3276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:04:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:04:57.286141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:57.367861 3289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:57.368091 3289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:57.368114 3289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:57.369380 3289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:04:57.369434 3289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:57.369447 3289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:57.369458 3289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:04:57.369469 3289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:04:57.371120 3289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:57.371390 3289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:57.371611 3289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:57.376754 3295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:57.376798 3294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:57.376945 3289 server_base.cc:1034] running on GCE node
W20260926 01:04:57.377012 3297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:57.377247 3289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:57.377584 3289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:57.378741 3289 hybrid_clock.cc:648] HybridClock initialized: now 1790384697378716 us; error 33 us; skew 500 ppm
Sep 26 01:04:57 dist-test-slave-l73v krb5kdc[3276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:57.387622 3289 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:04:57.388167 3289 webserver.cc:460] Webserver started at http://127.0.98.62:42155/ using document root <none> and password file <none>
I20260926 01:04:57.388370 3289 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:57.388419 3289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:57.388599 3289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:57.389698 3289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "9539cb15c86448f6b7a3b4619b05f74f"
format_stamp: "Formatted at 2026-09-26 01:04:57 on dist-test-slave-l73v"
server_key: "15f409545bef5965bb12d796aaaee30a"
server_key_iv: "5bf1916c4042ff1c0ecc5221413f4bfd"
server_key_version: "encryptionkey@0"
I20260926 01:04:57.390036 3289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "9539cb15c86448f6b7a3b4619b05f74f"
format_stamp: "Formatted at 2026-09-26 01:04:57 on dist-test-slave-l73v"
server_key: "15f409545bef5965bb12d796aaaee30a"
server_key_iv: "5bf1916c4042ff1c0ecc5221413f4bfd"
server_key_version: "encryptionkey@0"
I20260926 01:04:57.391463 3289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:04:57.392295 3304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:57.392532 3289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:04:57.392630 3289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "9539cb15c86448f6b7a3b4619b05f74f"
format_stamp: "Formatted at 2026-09-26 01:04:57 on dist-test-slave-l73v"
server_key: "15f409545bef5965bb12d796aaaee30a"
server_key_iv: "5bf1916c4042ff1c0ecc5221413f4bfd"
server_key_version: "encryptionkey@0"
I20260926 01:04:57.392696 3289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:57.404874 3289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:57.407099 3289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:57.407245 3289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:57.411201 3289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38011
I20260926 01:04:57.411244 3356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38011 every 8 connection(s)
I20260926 01:04:57.411584 3289 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:04:57.412428 3357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:57.415031 3357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f: Bootstrap starting.
I20260926 01:04:57.415645 3357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:57.415918 3357 log.cc:824] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:57.416736 3357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f: No bootstrap required, opened a new log
I20260926 01:04:57.418478 3357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539cb15c86448f6b7a3b4619b05f74f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } }
I20260926 01:04:57.418646 3357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:57.418668 3357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9539cb15c86448f6b7a3b4619b05f74f, State: Initialized, Role: FOLLOWER
I20260926 01:04:57.418807 3357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539cb15c86448f6b7a3b4619b05f74f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } }
I20260926 01:04:57.418883 3357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:04:57.418908 3357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:04:57.418941 3357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:57.419898 3357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539cb15c86448f6b7a3b4619b05f74f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } }
I20260926 01:04:57.420004 3357 leader_election.cc:302] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9539cb15c86448f6b7a3b4619b05f74f; no voters:
I20260926 01:04:57.420181 3357 leader_election.cc:288] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:04:57.420266 3360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:57.420449 3360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [term 1 LEADER]: Becoming Leader. State: Replica: 9539cb15c86448f6b7a3b4619b05f74f, State: Running, Role: LEADER
I20260926 01:04:57.420459 3357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:04:57.420572 3360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539cb15c86448f6b7a3b4619b05f74f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } }
I20260926 01:04:57.421106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 3289
I20260926 01:04:57.421136 3361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9539cb15c86448f6b7a3b4619b05f74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539cb15c86448f6b7a3b4619b05f74f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } } }
I20260926 01:04:57.421195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:04:57.421219 3361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:57.421384 392 external_mini_cluster.cc:1428] Setting key 3fde237e71c5734f9138fdbc8084c920
I20260926 01:04:57.421555 3369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:04:57.422086 3369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:04:57.422482 3362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9539cb15c86448f6b7a3b4619b05f74f. Latest consensus state: current_term: 1 leader_uuid: "9539cb15c86448f6b7a3b4619b05f74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9539cb15c86448f6b7a3b4619b05f74f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } } }
I20260926 01:04:57.422564 3362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:57.423846 3369 catalog_manager.cc:1347] Generated new cluster ID: 5d7abcdd9d644fe781ab9b6282be18ae
I20260926 01:04:57.423897 3369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:04:57 dist-test-slave-l73v krb5kdc[3276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:57.434064 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:57.421998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.0.98.62:38011)
0926 01:04:57.422267 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:57.422277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:57.422310 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:04:57.422750 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:57.422760 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:57.422827 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:57.422969 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:57.423713 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:57.425178 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:57.426289 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:57.427024 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:57.427158 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:57.430209 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:57.430214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:57.430215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:57.430230 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:57.432251 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:57.432756 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:57.432759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:57.432759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:57.432867 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:57.433085 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:57.433085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:57.433086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:57.433378 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:57.433521 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:57.433657 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 01:04:57.433792 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260926 01:04:57.435015 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:04:57.443514 3369 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:04:57.444243 3369 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:04:57.448122 3369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9539cb15c86448f6b7a3b4619b05f74f: Generated new TSK 0
I20260926 01:04:57.448324 3369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:04:57.451937 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:57.448373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.0.98.62:38011)
0926 01:04:57.448416 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:57.448418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:57.448643 (+ 225us) server_negotiation.cc:407] Connection header received
0926 01:04:57.448713 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:57.448714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:57.448724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:57.448779 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:57.449165 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:57.449697 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:57.450473 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:57.450701 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:57.450753 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:57.451512 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:57.451513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:57.451513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:04:57.451693 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:57.451710 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:04:57.451717 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:57.451762 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:57.451834 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 01:04:57.451857 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:04:57.454614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 3289
2026-09-26T01:04:57Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1255 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:04:57 dist-test-slave-l73v krb5kdc[3390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:04:57 dist-test-slave-l73v krb5kdc[3390](info): set up 2 sockets
Sep 26 01:04:57 dist-test-slave-l73v krb5kdc[3390](info): commencing operation
krb5kdc: starting...
W20260926 01:04:58.446589 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:04:58 dist-test-slave-l73v krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:04:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:04:58.578310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:58.664566 3403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:58.664870 3403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:58.664911 3403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:58.666318 3403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:04:58.666419 3403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:58.666446 3403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:58.666467 3403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:04:58.666495 3403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:04:58.668517 3403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:58.668825 3403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:58.669067 3403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:58.674139 3408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:58.674253 3411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:58.674144 3403 server_base.cc:1034] running on GCE node
W20260926 01:04:58.674144 3409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:58.674762 3403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:58.675077 3403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:58.676246 3403 hybrid_clock.cc:648] HybridClock initialized: now 1790384698676229 us; error 28 us; skew 500 ppm
Sep 26 01:04:58 dist-test-slave-l73v krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:58.678872 3403 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:04:58.679351 3403 webserver.cc:460] Webserver started at http://127.0.98.62:36793/ using document root <none> and password file <none>
I20260926 01:04:58.679548 3403 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:58.679598 3403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:58.679736 3403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:58.680852 3403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "d281a6bc39b24596b345d1b4d499e780"
format_stamp: "Formatted at 2026-09-26 01:04:58 on dist-test-slave-l73v"
server_key: "7a026f7ffdefe70cd54f9dc05620231d"
server_key_iv: "ee5c4a21eecddcd2d28a3ecefb86e6f4"
server_key_version: "encryptionkey@0"
I20260926 01:04:58.681202 3403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "d281a6bc39b24596b345d1b4d499e780"
format_stamp: "Formatted at 2026-09-26 01:04:58 on dist-test-slave-l73v"
server_key: "7a026f7ffdefe70cd54f9dc05620231d"
server_key_iv: "ee5c4a21eecddcd2d28a3ecefb86e6f4"
server_key_version: "encryptionkey@0"
I20260926 01:04:58.682596 3403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 01:04:58.683482 3418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:58.683647 3403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:58.683746 3403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "d281a6bc39b24596b345d1b4d499e780"
format_stamp: "Formatted at 2026-09-26 01:04:58 on dist-test-slave-l73v"
server_key: "7a026f7ffdefe70cd54f9dc05620231d"
server_key_iv: "ee5c4a21eecddcd2d28a3ecefb86e6f4"
server_key_version: "encryptionkey@0"
I20260926 01:04:58.683820 3403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:58.702369 3403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:58.704924 3403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:58.705080 3403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:58.709472 3403 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36447
I20260926 01:04:58.709568 3470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36447 every 8 connection(s)
I20260926 01:04:58.709906 3403 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:04:58.710934 3471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:04:58.713369 3471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780: Bootstrap starting.
I20260926 01:04:58.713440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 3403
I20260926 01:04:58.713536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:04:58.713706 392 external_mini_cluster.cc:1428] Setting key 50284555d7c5cd26ff65b7ea7c0a0937
I20260926 01:04:58.714095 3471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780: Neither blocks nor log segments found. Creating new log.
I20260926 01:04:58.714452 3471 log.cc:824] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780: Log is configured to *not* fsync() on all Append() calls
I20260926 01:04:58.715098 3471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780: No bootstrap required, opened a new log
I20260926 01:04:58.717157 3471 raft_consensus.cc:348] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d281a6bc39b24596b345d1b4d499e780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260926 01:04:58.717345 3471 raft_consensus.cc:374] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:04:58.717383 3471 raft_consensus.cc:729] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d281a6bc39b24596b345d1b4d499e780, State: Initialized, Role: FOLLOWER
I20260926 01:04:58.717506 3471 consensus_queue.cc:260] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d281a6bc39b24596b345d1b4d499e780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260926 01:04:58.717592 3471 raft_consensus.cc:388] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:04:58.717626 3471 raft_consensus.cc:482] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:04:58.717667 3471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:04:58.718304 3471 raft_consensus.cc:504] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d281a6bc39b24596b345d1b4d499e780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260926 01:04:58.718433 3471 leader_election.cc:302] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d281a6bc39b24596b345d1b4d499e780; no voters:
I20260926 01:04:58.718600 3471 leader_election.cc:288] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:04:58.718676 3476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:04:58.718796 3476 raft_consensus.cc:686] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [term 1 LEADER]: Becoming Leader. State: Replica: d281a6bc39b24596b345d1b4d499e780, State: Running, Role: LEADER
I20260926 01:04:58.718976 3471 sys_catalog.cc:564] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:04:58.719378 3476 consensus_queue.cc:237] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d281a6bc39b24596b345d1b4d499e780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260926 01:04:58.719796 3477 sys_catalog.cc:455] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d281a6bc39b24596b345d1b4d499e780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d281a6bc39b24596b345d1b4d499e780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } } }
I20260926 01:04:58.719888 3477 sys_catalog.cc:458] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:58.720221 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d281a6bc39b24596b345d1b4d499e780. Latest consensus state: current_term: 1 leader_uuid: "d281a6bc39b24596b345d1b4d499e780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d281a6bc39b24596b345d1b4d499e780" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } } }
I20260926 01:04:58.720292 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780 [sys.catalog]: This master's current role is: LEADER
I20260926 01:04:58.720722 3485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 01:04:58 dist-test-slave-l73v krb5kdc[3390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:58.721268 3485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:04:58.722975 3485 catalog_manager.cc:1347] Generated new cluster ID: 40506c687844418bb778bd72d0d7aabb
I20260926 01:04:58.723027 3485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:04:58.725428 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:58.714293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.0.98.62:36447)
0926 01:04:58.714534 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:58.714543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:58.714579 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:04:58.715058 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:58.715065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:58.715117 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:58.715249 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:58.716027 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:58.716913 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:58.717981 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:58.718468 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:58.718626 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:58.721560 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:58.721564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:58.721565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:58.721578 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:58.723687 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:58.724151 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:58.724153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:58.724154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:58.724265 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:58.724487 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:58.724488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:58.724488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:58.724772 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:58.724901 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:58.725067 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 01:04:58.725209 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260926 01:04:58.733608 3485 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:04:58.734225 3485 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:04:58.745306 3485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d281a6bc39b24596b345d1b4d499e780: Generated new TSK 0
I20260926 01:04:58.745522 3485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:04:58.781796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:58.870927 3497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:58.871156 3497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:58.871193 3497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:58.872565 3497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:58.872620 3497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:58.872661 3497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:04:58.874610 3497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:58.874957 3497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:58.875203 3497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:58.880733 3502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:58.880733 3505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:58.880767 3497 server_base.cc:1034] running on GCE node
W20260926 01:04:58.880733 3503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:58.881155 3497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:58.881389 3497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:58.882552 3497 hybrid_clock.cc:648] HybridClock initialized: now 1790384698882525 us; error 42 us; skew 500 ppm
Sep 26 01:04:58 dist-test-slave-l73v krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:58.885288 3497 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:04:58.885802 3497 webserver.cc:460] Webserver started at http://127.0.98.1:46405/ using document root <none> and password file <none>
I20260926 01:04:58.886034 3497 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:58.886087 3497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:58.886219 3497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:58.887471 3497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "eaa9941de9e14a56a3d9e52a89d70f9c"
format_stamp: "Formatted at 2026-09-26 01:04:58 on dist-test-slave-l73v"
server_key: "5f7a756f5f2bf89833f5d3eb368de329"
server_key_iv: "d746de19f6aa034f0cb24156985006eb"
server_key_version: "encryptionkey@0"
I20260926 01:04:58.887847 3497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eaa9941de9e14a56a3d9e52a89d70f9c"
format_stamp: "Formatted at 2026-09-26 01:04:58 on dist-test-slave-l73v"
server_key: "5f7a756f5f2bf89833f5d3eb368de329"
server_key_iv: "d746de19f6aa034f0cb24156985006eb"
server_key_version: "encryptionkey@0"
I20260926 01:04:58.889505 3497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:04:58.891235 3512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:58.891798 3497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:58.891904 3497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "eaa9941de9e14a56a3d9e52a89d70f9c"
format_stamp: "Formatted at 2026-09-26 01:04:58 on dist-test-slave-l73v"
server_key: "5f7a756f5f2bf89833f5d3eb368de329"
server_key_iv: "d746de19f6aa034f0cb24156985006eb"
server_key_version: "encryptionkey@0"
I20260926 01:04:58.891983 3497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:58.928860 3497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:58.931131 3497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:58.931293 3497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:58.931521 3497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:58.931836 3497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:58.931869 3497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:58.931900 3497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:58.931923 3497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:58.937377 3497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38483
I20260926 01:04:58.937464 3625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38483 every 8 connection(s)
I20260926 01:04:58.937798 3497 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:04:58 dist-test-slave-l73v krb5kdc[3390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:58.947508 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:58.938633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32855 (local address 127.0.98.62:36447)
0926 01:04:58.938746 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:58.938748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:58.938965 (+ 217us) server_negotiation.cc:407] Connection header received
0926 01:04:58.939424 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:58.939425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:58.939436 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:58.939534 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:58.940456 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:58.941002 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:58.942070 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:58.942291 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:58.942356 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:58.944680 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:58.944682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:58.944682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:58.944690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:58.946014 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:58.946506 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:58.946508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:58.946509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:58.946582 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:58.946847 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:58.946848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:58.946849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:58.946979 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:58.947047 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:58.947327 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 01:04:58.947406 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 01:04:58.947582 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:58.938590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.1:32855)
0926 01:04:58.938819 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:58.938857 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:58.939330 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:58.939591 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:58.939594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:58.940154 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:58.940355 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:58.940360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:58.941093 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:58.941094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:58.941983 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:58.941985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:58.942421 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:58.942423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:58.942543 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:58.943090 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:58.943100 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:58.944558 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:58.946115 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:58.946123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:58.946125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:58.946375 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:58.946655 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:58.946655 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:58.946656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:58.946769 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:58.947041 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:58.947045 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:58.947176 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 01:04:58.947263 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 01:04:58.947395 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260926 01:04:58.947932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 3497
I20260926 01:04:58.948014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:04:58.948233 392 external_mini_cluster.cc:1428] Setting key 75505f457501d2b219dff9c11ca7c903
I20260926 01:04:58.948205 3626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260926 01:04:58.948302 3626 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:58.948513 3626 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260926 01:04:58.949257 3435 ts_manager.cc:194] Registered new tserver with Master: eaa9941de9e14a56a3d9e52a89d70f9c (127.0.98.1:38483)
I20260926 01:04:58.950117 3435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32855
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:04:58.974582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:59.061205 3631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:59.061450 3631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:59.061480 3631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:59.062772 3631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:59.062831 3631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:59.062863 3631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:04:59.064433 3631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:59.064664 3631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:59.064867 3631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:59.070080 3639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:59.070080 3637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:59.070096 3636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:59.070305 3631 server_base.cc:1034] running on GCE node
I20260926 01:04:59.070521 3631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:59.070806 3631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:59.071952 3631 hybrid_clock.cc:648] HybridClock initialized: now 1790384699071941 us; error 33 us; skew 500 ppm
Sep 26 01:04:59 dist-test-slave-l73v krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:59.074658 3631 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:04:59.075099 3631 webserver.cc:460] Webserver started at http://127.0.98.2:32845/ using document root <none> and password file <none>
I20260926 01:04:59.075305 3631 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:59.075350 3631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:59.075469 3631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:59.076538 3631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "c9cd31c991af4471b4d941efa8d5ee56"
format_stamp: "Formatted at 2026-09-26 01:04:59 on dist-test-slave-l73v"
server_key: "cc24105f3582ce51ec68c44bb7ade33b"
server_key_iv: "1f586bf43394c40b8d2fb916087e7eee"
server_key_version: "encryptionkey@0"
I20260926 01:04:59.076897 3631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c9cd31c991af4471b4d941efa8d5ee56"
format_stamp: "Formatted at 2026-09-26 01:04:59 on dist-test-slave-l73v"
server_key: "cc24105f3582ce51ec68c44bb7ade33b"
server_key_iv: "1f586bf43394c40b8d2fb916087e7eee"
server_key_version: "encryptionkey@0"
I20260926 01:04:59.078414 3631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:04:59.079252 3646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:59.079402 3631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:59.079506 3631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "c9cd31c991af4471b4d941efa8d5ee56"
format_stamp: "Formatted at 2026-09-26 01:04:59 on dist-test-slave-l73v"
server_key: "cc24105f3582ce51ec68c44bb7ade33b"
server_key_iv: "1f586bf43394c40b8d2fb916087e7eee"
server_key_version: "encryptionkey@0"
I20260926 01:04:59.079574 3631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:59.120852 3631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:59.123086 3631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:59.123343 3631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:59.123591 3631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:59.123958 3631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:59.123994 3631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:59.124015 3631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:59.124027 3631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:59.129595 3631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38623
I20260926 01:04:59.129729 3759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38623 every 8 connection(s)
I20260926 01:04:59.129972 3631 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:04:59.130227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 3631
I20260926 01:04:59.130334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:04:59.130538 392 external_mini_cluster.cc:1428] Setting key e60e3a751fa8e47bc642ee619d87c911
Sep 26 01:04:59 dist-test-slave-l73v krb5kdc[3390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:04:59.140542 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:59.130696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44469 (local address 127.0.98.62:36447)
0926 01:04:59.130852 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:59.130854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:59.131000 (+ 146us) server_negotiation.cc:407] Connection header received
0926 01:04:59.131574 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:59.131575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:59.131588 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:59.131643 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:59.132797 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:59.133300 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:59.134790 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:59.134979 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:59.135036 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:59.137692 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:59.137694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:59.137695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:59.137702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:59.139144 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:59.139567 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:59.139569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:59.139569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:59.139620 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:59.139890 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:59.139891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:59.139891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:59.140003 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:59.140064 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:59.140391 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 01:04:59.140466 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260926 01:04:59.140722 3762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:59.130687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.2:44469)
0926 01:04:59.130867 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:59.130918 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:59.131473 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:59.131723 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:59.131728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:59.132374 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:59.132684 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:59.132692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:59.133403 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:59.133404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:59.134708 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:59.134711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:59.135064 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:59.135065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:59.135223 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:59.135829 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:59.135841 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:59.137543 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:59.139238 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:59.139248 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:59.139251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:59.139483 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:59.139708 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:59.139709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:59.139709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:59.139826 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:59.140076 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:59.140080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:59.140262 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 01:04:59.140353 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 01:04:59.140491 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 01:04:59.141227 3760 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260926 01:04:59.141345 3760 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:59.141573 3760 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260926 01:04:59.141970 3435 ts_manager.cc:194] Registered new tserver with Master: c9cd31c991af4471b4d941efa8d5ee56 (127.0.98.2:38623)
I20260926 01:04:59.142467 3435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:04:59.158775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:04:59.245931 3765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:04:59.246171 3765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:04:59.246199 3765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:04:59.247475 3765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:04:59.247534 3765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:04:59.247565 3765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:04:59.249017 3765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:04:59.249228 3765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:04:59.249435 3765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:04:59.254624 3770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:04:59.254647 3773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:59.254770 3765 server_base.cc:1034] running on GCE node
W20260926 01:04:59.254591 3771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:04:59.255093 3765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:04:59.255373 3765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:04:59.256556 3765 hybrid_clock.cc:648] HybridClock initialized: now 1790384699256543 us; error 28 us; skew 500 ppm
Sep 26 01:04:59 dist-test-slave-l73v krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:04:59.259385 3765 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:04:59.259825 3765 webserver.cc:460] Webserver started at http://127.0.98.3:38567/ using document root <none> and password file <none>
I20260926 01:04:59.260031 3765 fs_manager.cc:359] Metadata directory not provided
I20260926 01:04:59.260079 3765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:04:59.260286 3765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:04:59.261375 3765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "1f3a1b9ec645496ca9dee56c564e3b29"
format_stamp: "Formatted at 2026-09-26 01:04:59 on dist-test-slave-l73v"
server_key: "7c79a9773e77294dcb012f77305e85de"
server_key_iv: "ec851e395ac1b78419ae27fa50cbbcd1"
server_key_version: "encryptionkey@0"
I20260926 01:04:59.261718 3765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1f3a1b9ec645496ca9dee56c564e3b29"
format_stamp: "Formatted at 2026-09-26 01:04:59 on dist-test-slave-l73v"
server_key: "7c79a9773e77294dcb012f77305e85de"
server_key_iv: "ec851e395ac1b78419ae27fa50cbbcd1"
server_key_version: "encryptionkey@0"
I20260926 01:04:59.263104 3765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:04:59.263995 3780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:59.264268 3765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:04:59.264371 3765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "1f3a1b9ec645496ca9dee56c564e3b29"
format_stamp: "Formatted at 2026-09-26 01:04:59 on dist-test-slave-l73v"
server_key: "7c79a9773e77294dcb012f77305e85de"
server_key_iv: "ec851e395ac1b78419ae27fa50cbbcd1"
server_key_version: "encryptionkey@0"
I20260926 01:04:59.264446 3765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:04:59.278932 3765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:04:59.281095 3765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:04:59.281243 3765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:04:59.281478 3765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:04:59.281762 3765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:04:59.281795 3765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:59.281826 3765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:04:59.281848 3765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:04:59.287635 3765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36197
I20260926 01:04:59.287695 3893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36197 every 8 connection(s)
I20260926 01:04:59.288000 3765 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:04:59.293905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 3765
I20260926 01:04:59.293977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:04:59.294142 392 external_mini_cluster.cc:1428] Setting key 5653835d145d0367e12b055d1a74aff4
Sep 26 01:04:59 dist-test-slave-l73v krb5kdc[3390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:59.297917 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:59.288717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40719 (local address 127.0.98.62:36447)
0926 01:04:59.288793 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:59.288795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:59.288982 (+ 187us) server_negotiation.cc:407] Connection header received
0926 01:04:59.289462 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:59.289464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:59.289474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:59.289540 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:59.290549 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:59.291117 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:59.292105 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:59.292301 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:59.292355 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:59.294971 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:59.294973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:59.294973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:59.294982 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:59.296492 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:59.296914 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:59.296914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:59.296915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:59.296967 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:59.297292 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:59.297293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:59.297293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:59.297418 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:59.297475 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:59.297775 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 01:04:59.297848 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 01:04:59.298164 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:59.288670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.3:40719)
0926 01:04:59.288839 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 01:04:59.288883 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:04:59.289359 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:04:59.289614 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:04:59.289617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:04:59.290195 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:04:59.290437 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:59.290444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:59.291207 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:59.291208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:59.292024 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:04:59.292027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:04:59.292398 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:04:59.292400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:04:59.292538 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:59.293165 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:04:59.293173 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:04:59.294859 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:04:59.296590 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:59.296598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:59.296601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:59.296844 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:59.297081 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:04:59.297082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:04:59.297082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:04:59.297213 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:04:59.297501 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:04:59.297507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:04:59.297658 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 01:04:59.297757 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 01:04:59.297895 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260926 01:04:59.298612 3894 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260926 01:04:59.298713 3894 heartbeater.cc:464] Registering TS with master...
I20260926 01:04:59.298868 3894 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260926 01:04:59.299270 3435 ts_manager.cc:194] Registered new tserver with Master: 1f3a1b9ec645496ca9dee56c564e3b29 (127.0.98.3:36197)
I20260926 01:04:59.299651 3435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40719
I20260926 01:04:59.306355 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 01:04:59 dist-test-slave-l73v krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384699, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 01:04:59 dist-test-slave-l73v krb5kdc[3390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384699, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:04:59.328263 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:04:59.321193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.0.98.62:36447)
0926 01:04:59.321252 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:04:59.321253 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:04:59.321322 (+ 69us) server_negotiation.cc:407] Connection header received
0926 01:04:59.321384 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:04:59.321385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:04:59.321396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:04:59.321444 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:04:59.322017 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:59.322557 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:59.323274 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:04:59.323465 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:04:59.323519 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:04:59.325781 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:04:59.325783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:04:59.325784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:04:59.325791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:04:59.327197 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:59.327512 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:59.327513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:59.327514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:59.327565 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:04:59.327727 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:04:59.327727 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:04:59.327728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:04:59.327880 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:04:59.327975 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 01:04:59.328084 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 01:04:59.328164 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 01:04:59.328521 3435 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45980
I20260926 01:04:59.328953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 3497
I20260926 01:04:59.332913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 3631
I20260926 01:04:59.336697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 3765
I20260926 01:04:59.340641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 3403
2026-09-26T01:04:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1887 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:04:59 dist-test-slave-l73v krb5kdc[3906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:04:59 dist-test-slave-l73v krb5kdc[3906](info): set up 2 sockets
Sep 26 01:04:59 dist-test-slave-l73v krb5kdc[3906](info): commencing operation
krb5kdc: starting...
W20260926 01:05:00.306330 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:05:00 dist-test-slave-l73v krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:05:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:05:00.427557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45519
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:00.509418 3919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:00.509665 3919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:00.509692 3919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:00.511006 3919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:00.511063 3919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:00.511080 3919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:00.511093 3919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:00.511106 3919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:00.512737 3919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:00.512995 3919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:00.513233 3919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:00.518209 3924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:00.518268 3919 server_base.cc:1034] running on GCE node
W20260926 01:05:00.518190 3925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:00.518210 3927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:00.518613 3919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:00.518931 3919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:00.520107 3919 hybrid_clock.cc:648] HybridClock initialized: now 1790384700520076 us; error 44 us; skew 500 ppm
Sep 26 01:05:00 dist-test-slave-l73v krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:00.522604 3919 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:05:00.523027 3919 webserver.cc:460] Webserver started at http://127.0.98.62:35567/ using document root <none> and password file <none>
I20260926 01:05:00.523370 3919 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:00.523447 3919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:00.523662 3919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:00.524768 3919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "d5bd19b5698a468c813ed07a3ea83039"
format_stamp: "Formatted at 2026-09-26 01:05:00 on dist-test-slave-l73v"
server_key: "d2e6d75880672edf2d80ce1bf0e80871"
server_key_iv: "9e74da6007c3491de43d03291e051360"
server_key_version: "encryptionkey@0"
I20260926 01:05:00.525098 3919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "d5bd19b5698a468c813ed07a3ea83039"
format_stamp: "Formatted at 2026-09-26 01:05:00 on dist-test-slave-l73v"
server_key: "d2e6d75880672edf2d80ce1bf0e80871"
server_key_iv: "9e74da6007c3491de43d03291e051360"
server_key_version: "encryptionkey@0"
I20260926 01:05:00.526517 3919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:05:00.527290 3934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:00.527494 3919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:00.527585 3919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "d5bd19b5698a468c813ed07a3ea83039"
format_stamp: "Formatted at 2026-09-26 01:05:00 on dist-test-slave-l73v"
server_key: "d2e6d75880672edf2d80ce1bf0e80871"
server_key_iv: "9e74da6007c3491de43d03291e051360"
server_key_version: "encryptionkey@0"
I20260926 01:05:00.527654 3919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:00.538828 3919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:00.541153 3919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:00.541333 3919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:00.545358 3919 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45519
I20260926 01:05:00.545444 3986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45519 every 8 connection(s)
I20260926 01:05:00.545774 3919 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:00.546703 3987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:00.549050 3987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039: Bootstrap starting.
I20260926 01:05:00.549624 3987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:00.549872 3987 log.cc:824] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:00.550663 3987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039: No bootstrap required, opened a new log
I20260926 01:05:00.552397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 3919
I20260926 01:05:00.552498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:00.552345 3987 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bd19b5698a468c813ed07a3ea83039" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } }
I20260926 01:05:00.552531 3987 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:00.552572 3987 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5bd19b5698a468c813ed07a3ea83039, State: Initialized, Role: FOLLOWER
I20260926 01:05:00.552675 392 external_mini_cluster.cc:1428] Setting key f8ccfd72aa4d04f507aae431dac2225b
I20260926 01:05:00.552729 3987 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bd19b5698a468c813ed07a3ea83039" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } }
I20260926 01:05:00.552809 3987 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:00.552840 3987 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:00.552888 3987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:00.553992 3987 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bd19b5698a468c813ed07a3ea83039" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } }
I20260926 01:05:00.554090 3987 leader_election.cc:302] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5bd19b5698a468c813ed07a3ea83039; no voters:
I20260926 01:05:00.554250 3987 leader_election.cc:288] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:00.554291 3992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:00.554436 3992 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [term 1 LEADER]: Becoming Leader. State: Replica: d5bd19b5698a468c813ed07a3ea83039, State: Running, Role: LEADER
I20260926 01:05:00.554527 3992 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bd19b5698a468c813ed07a3ea83039" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } }
I20260926 01:05:00.554581 3987 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:00.554873 3992 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5bd19b5698a468c813ed07a3ea83039. Latest consensus state: current_term: 1 leader_uuid: "d5bd19b5698a468c813ed07a3ea83039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bd19b5698a468c813ed07a3ea83039" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } } }
I20260926 01:05:00.554948 3992 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:00.555102 3994 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5bd19b5698a468c813ed07a3ea83039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bd19b5698a468c813ed07a3ea83039" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } } }
I20260926 01:05:00.555176 3994 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:00.555297 3998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:00.555827 3998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:00.558832 3998 catalog_manager.cc:1347] Generated new cluster ID: 2133c6b4f3e54e1c83f782da6951aca0
I20260926 01:05:00.558903 3998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:05:00 dist-test-slave-l73v krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:00.563202 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:00.553157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.0.98.62:45519)
0926 01:05:00.553386 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:00.553398 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:00.553438 (+ 40us) server_negotiation.cc:407] Connection header received
0926 01:05:00.553589 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:00.553597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:00.553664 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:00.553828 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:00.554641 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:00.556063 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:00.557008 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:00.557453 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:00.557553 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:00.560122 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:00.560127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:00.560127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:00.560139 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:00.561644 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:00.562050 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:00.562054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:00.562054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:00.562148 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:00.562328 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:00.562329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:00.562329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:00.562634 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:00.562779 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:00.562890 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 01:05:00.563032 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":167,"thread_start_us":112,"threads_started":1}
I20260926 01:05:00.574898 3998 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:00.575649 3998 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:00.579949 3998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5bd19b5698a468c813ed07a3ea83039: Generated new TSK 0
I20260926 01:05:00.580210 3998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:05:00.611407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45519
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:00.694078 4013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:00.694350 4013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:00.694424 4013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:00.696060 4013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:00.696151 4013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:00.696195 4013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:00.698258 4013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:00.698590 4013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:00.698832 4013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:00.704092 4021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:00.704094 4018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:00.704066 4019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:00.704560 4013 server_base.cc:1034] running on GCE node
I20260926 01:05:00.704916 4013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:00.705252 4013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:00.706471 4013 hybrid_clock.cc:648] HybridClock initialized: now 1790384700706374 us; error 112 us; skew 500 ppm
Sep 26 01:05:00 dist-test-slave-l73v krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:00.709091 4013 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:05:00.709540 4013 webserver.cc:460] Webserver started at http://127.0.98.1:34105/ using document root <none> and password file <none>
I20260926 01:05:00.709761 4013 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:00.709806 4013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:00.709921 4013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:00.711057 4013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "c4b4a638f3c74bc8861ec22197836859"
format_stamp: "Formatted at 2026-09-26 01:05:00 on dist-test-slave-l73v"
server_key: "6da132c3d2afe3e790dcebeebe069565"
server_key_iv: "4d5b82b28c98925dbbc2830658481a5c"
server_key_version: "encryptionkey@0"
I20260926 01:05:00.711395 4013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c4b4a638f3c74bc8861ec22197836859"
format_stamp: "Formatted at 2026-09-26 01:05:00 on dist-test-slave-l73v"
server_key: "6da132c3d2afe3e790dcebeebe069565"
server_key_iv: "4d5b82b28c98925dbbc2830658481a5c"
server_key_version: "encryptionkey@0"
I20260926 01:05:00.712973 4013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:00.713876 4028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:00.714083 4013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:00.714183 4013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "c4b4a638f3c74bc8861ec22197836859"
format_stamp: "Formatted at 2026-09-26 01:05:00 on dist-test-slave-l73v"
server_key: "6da132c3d2afe3e790dcebeebe069565"
server_key_iv: "4d5b82b28c98925dbbc2830658481a5c"
server_key_version: "encryptionkey@0"
I20260926 01:05:00.714247 4013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:00.724766 4013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:00.727227 4013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:00.727370 4013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:00.727739 4013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:00.728178 4013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:00.728237 4013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:00.728268 4013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:00.728284 4013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:00.736253 4013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45637
I20260926 01:05:00.736346 4141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45637 every 8 connection(s)
I20260926 01:05:00.736670 4013 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:05:00.737042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 4013
I20260926 01:05:00.737125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:05:00.737295 392 external_mini_cluster.cc:1428] Setting key 478b18e9f885c9cdbaf6c1c4942cbf4f
Sep 26 01:05:00 dist-test-slave-l73v krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:05:00.747809 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:00.737459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53041 (local address 127.0.98.62:45519)
0926 01:05:00.737513 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:00.737515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:00.737745 (+ 230us) server_negotiation.cc:407] Connection header received
0926 01:05:00.738996 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:00.738998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:00.739011 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:00.739077 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:00.739808 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:00.740378 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:00.741487 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:00.741710 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:00.741770 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:00.744162 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:00.744163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:00.744164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:00.744172 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:00.746181 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:00.746716 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:00.746718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:00.746718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:00.746789 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:00.747043 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:00.747044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:00.747044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:00.747168 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:00.747224 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:00.747598 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 01:05:00.747701 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:05:00.747991 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:00.737426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45519 (local address 127.0.98.1:53041)
0926 01:05:00.737618 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:00.737670 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:00.738852 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:00.739148 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:00.739151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:00.739431 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:00.739647 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:00.739652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:00.740507 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:00.740508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:00.741382 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:00.741385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:00.741836 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:00.741837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:00.741954 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:00.742634 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:00.742643 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:00.744054 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:00.746333 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:00.746342 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:00.746344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:00.746642 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:00.746852 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:00.746853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:00.746853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:00.746976 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:00.747271 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:00.747277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:00.747420 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 01:05:00.747547 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 01:05:00.747765 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 01:05:00.748520 4142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45519
I20260926 01:05:00.748624 4142 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:00.748847 4142 heartbeater.cc:511] Master 127.0.98.62:45519 requested a full tablet report, sending...
I20260926 01:05:00.749791 3951 ts_manager.cc:194] Registered new tserver with Master: c4b4a638f3c74bc8861ec22197836859 (127.0.98.1:45637)
I20260926 01:05:00.754908 3951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:05:00.770102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45519
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:00.852768 4147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:00.853067 4147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:00.853097 4147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:00.854683 4147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:00.854763 4147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:00.854796 4147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:00.856585 4147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:00.857069 4147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:00.857446 4147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:00.863540 4152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:00.863543 4153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:00.863543 4155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:00.863662 4147 server_base.cc:1034] running on GCE node
I20260926 01:05:00.864125 4147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:00.864437 4147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:00.865612 4147 hybrid_clock.cc:648] HybridClock initialized: now 1790384700865596 us; error 38 us; skew 500 ppm
Sep 26 01:05:00 dist-test-slave-l73v krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:00.868366 4147 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:05:00.868840 4147 webserver.cc:460] Webserver started at http://127.0.98.2:41837/ using document root <none> and password file <none>
I20260926 01:05:00.869043 4147 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:00.869091 4147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:00.869359 4147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:00.870553 4147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "1ec89a4a07e54e4498007b80463bb154"
format_stamp: "Formatted at 2026-09-26 01:05:00 on dist-test-slave-l73v"
server_key: "d18254551d9328a358412a700f3bbbb9"
server_key_iv: "552aafb1c17a9e7767ce278cf16efce8"
server_key_version: "encryptionkey@0"
I20260926 01:05:00.870944 4147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1ec89a4a07e54e4498007b80463bb154"
format_stamp: "Formatted at 2026-09-26 01:05:00 on dist-test-slave-l73v"
server_key: "d18254551d9328a358412a700f3bbbb9"
server_key_iv: "552aafb1c17a9e7767ce278cf16efce8"
server_key_version: "encryptionkey@0"
I20260926 01:05:00.872359 4147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:05:00.873231 4162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:00.873634 4147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:00.873752 4147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "1ec89a4a07e54e4498007b80463bb154"
format_stamp: "Formatted at 2026-09-26 01:05:00 on dist-test-slave-l73v"
server_key: "d18254551d9328a358412a700f3bbbb9"
server_key_iv: "552aafb1c17a9e7767ce278cf16efce8"
server_key_version: "encryptionkey@0"
I20260926 01:05:00.873827 4147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:00.886840 4147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:00.889511 4147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:00.889693 4147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:00.889981 4147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:00.890329 4147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:00.890367 4147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:00.890462 4147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:00.890488 4147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:00.896549 4147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37539
I20260926 01:05:00.896615 4275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37539 every 8 connection(s)
I20260926 01:05:00.896939 4147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:05:00 dist-test-slave-l73v krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:00.906224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 4147
I20260926 01:05:00.906314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:05:00.906522 392 external_mini_cluster.cc:1428] Setting key fba87e7f37b90289726b005a25119193
I20260926 01:05:00.907225 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:00.897761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44157 (local address 127.0.98.62:45519)
0926 01:05:00.897830 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:00.897832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:00.898061 (+ 229us) server_negotiation.cc:407] Connection header received
0926 01:05:00.899247 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:00.899248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:00.899263 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:00.899315 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:00.899974 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:00.900506 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:00.901533 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:00.901736 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:00.901792 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:00.904298 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:00.904300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:00.904300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:00.904309 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:00.905680 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:00.906114 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:00.906115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:00.906116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:00.906171 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:00.906452 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:00.906453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:00.906453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:00.906577 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:00.906630 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:00.907050 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 01:05:00.907131 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:05:00.907418 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:00.897722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45519 (local address 127.0.98.2:44157)
0926 01:05:00.897936 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:00.897974 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:00.899102 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:00.899396 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:00.899400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:00.899672 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:00.899874 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:00.899879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:00.900592 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:00.900593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:00.901446 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:00.901449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:00.901810 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:00.901811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:00.901929 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:00.902676 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:00.902690 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:00.904184 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:00.905780 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:00.905789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:00.905792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:00.906039 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:00.906227 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:00.906228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:00.906228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:00.906371 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:00.906652 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:00.906660 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:00.906848 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 01:05:00.906978 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 01:05:00.907161 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":135,"threads_started":1}
I20260926 01:05:00.907923 4276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45519
I20260926 01:05:00.908026 4276 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:00.908214 4276 heartbeater.cc:511] Master 127.0.98.62:45519 requested a full tablet report, sending...
I20260926 01:05:00.908587 3951 ts_manager.cc:194] Registered new tserver with Master: 1ec89a4a07e54e4498007b80463bb154 (127.0.98.2:37539)
I20260926 01:05:00.909001 3951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44157
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:05:00.934079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45519
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:01.018664 4281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:01.018900 4281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:01.018926 4281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:01.020193 4281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:01.020238 4281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:01.020265 4281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:01.021826 4281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:01.022059 4281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:01.022269 4281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:01.027544 4287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:01.027539 4289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:01.027539 4286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:01.027838 4281 server_base.cc:1034] running on GCE node
I20260926 01:05:01.028049 4281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:01.028311 4281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:01.029614 4281 hybrid_clock.cc:648] HybridClock initialized: now 1790384701029599 us; error 26 us; skew 500 ppm
Sep 26 01:05:01 dist-test-slave-l73v krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:01.032382 4281 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:05:01.032877 4281 webserver.cc:460] Webserver started at http://127.0.98.3:40263/ using document root <none> and password file <none>
I20260926 01:05:01.033097 4281 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:01.033143 4281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:01.033250 4281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:01.034308 4281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "39fba49644714b69a257740b74975055"
format_stamp: "Formatted at 2026-09-26 01:05:01 on dist-test-slave-l73v"
server_key: "ca0c1aee10a14a75e360ac69311679b4"
server_key_iv: "4b13f8014613e70771be4a3e7b8bdd6b"
server_key_version: "encryptionkey@0"
I20260926 01:05:01.034705 4281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "39fba49644714b69a257740b74975055"
format_stamp: "Formatted at 2026-09-26 01:05:01 on dist-test-slave-l73v"
server_key: "ca0c1aee10a14a75e360ac69311679b4"
server_key_iv: "4b13f8014613e70771be4a3e7b8bdd6b"
server_key_version: "encryptionkey@0"
I20260926 01:05:01.036087 4281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:01.036964 4296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:01.037156 4281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:01.037245 4281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "39fba49644714b69a257740b74975055"
format_stamp: "Formatted at 2026-09-26 01:05:01 on dist-test-slave-l73v"
server_key: "ca0c1aee10a14a75e360ac69311679b4"
server_key_iv: "4b13f8014613e70771be4a3e7b8bdd6b"
server_key_version: "encryptionkey@0"
I20260926 01:05:01.037315 4281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:01.058015 4281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:01.060401 4281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:01.060535 4281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:01.060807 4281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:01.061120 4281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:01.061152 4281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:01.061173 4281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:01.061187 4281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:01.066939 4281 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34701
I20260926 01:05:01.067041 4409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34701 every 8 connection(s)
I20260926 01:05:01.067308 4281 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:05:01.069371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 4281
I20260926 01:05:01.069468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:05:01.069638 392 external_mini_cluster.cc:1428] Setting key e02630c43a8b605fc94a86431b3c539e
Sep 26 01:05:01 dist-test-slave-l73v krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:01.077621 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:01.068202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34585 (local address 127.0.98.62:45519)
0926 01:05:01.068254 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:01.068256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:01.068454 (+ 198us) server_negotiation.cc:407] Connection header received
0926 01:05:01.069468 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:01.069469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:01.069483 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:01.069528 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:01.070330 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:01.070930 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:01.071960 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:01.072152 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:01.072208 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:01.074477 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:01.074478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:01.074479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:01.074498 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:01.075991 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:01.076582 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:01.076583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:01.076583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:01.076643 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:01.076905 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:01.076906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:01.076907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:01.077040 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:01.077098 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:01.077419 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 01:05:01.077526 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:05:01.077816 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:01.068166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45519 (local address 127.0.98.3:34585)
0926 01:05:01.068316 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:01.068360 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:01.069346 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:01.069649 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:01.069652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:01.070000 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:01.070257 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:01.070263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:01.071024 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:01.071026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:01.071869 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:01.071871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:01.072222 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:01.072223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:01.072354 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:01.072890 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:01.072898 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:01.074326 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:01.076099 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:01.076110 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:01.076199 (+ 89us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:01.076461 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:01.076726 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:01.076727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:01.076727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:01.076844 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:01.077098 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:01.077105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:01.077291 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 01:05:01.077390 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 01:05:01.077577 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260926 01:05:01.078519 4410 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45519
I20260926 01:05:01.078619 4410 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:01.078786 4410 heartbeater.cc:511] Master 127.0.98.62:45519 requested a full tablet report, sending...
I20260926 01:05:01.079269 3951 ts_manager.cc:194] Registered new tserver with Master: 39fba49644714b69a257740b74975055 (127.0.98.3:34701)
I20260926 01:05:01.079766 3951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34585
I20260926 01:05:01.081837 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 01:05:01 dist-test-slave-l73v krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:05:01 dist-test-slave-l73v krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:05:01.106320 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:01.097338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.0.98.1:45637)
0926 01:05:01.097505 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:01.097511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:01.097524 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:05:01.097576 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:01.097578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:01.097594 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:01.097672 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:01.098328 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:01.099143 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:01.099914 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:01.100268 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:01.100345 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:01.102759 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:01.102771 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:01.102771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:01.102779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:01.104887 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:01.105322 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:01.105326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:01.105326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:01.105424 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:01.105624 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:01.105625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:01.105626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:01.105819 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:01.105990 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:01.106067 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 01:05:01.106164 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
Sep 26 01:05:01 dist-test-slave-l73v krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:01.115928 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:01.108054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.0.98.62:45519)
0926 01:05:01.108116 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:01.108118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:01.108126 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:01.108174 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:01.108175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:01.108184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:01.108226 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:01.108760 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:01.109256 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:01.110014 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:01.110219 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:01.110282 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:01.112712 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:01.112714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:01.112714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:01.112722 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:01.114031 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:01.114734 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:01.114735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:01.114736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:01.114813 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:01.115185 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:01.115187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:01.115187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:01.115428 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:01.115671 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:01.115726 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 01:05:01.115851 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 01:05:01.116266 3951 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37740
I20260926 01:05:01.116672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 4013
I20260926 01:05:01.121060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 4147
I20260926 01:05:01.125437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 4281
I20260926 01:05:01.129590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 3919
2026-09-26T01:05:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1789 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:05:01 dist-test-slave-l73v krb5kdc[4422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:05:01 dist-test-slave-l73v krb5kdc[4422](info): set up 2 sockets
Sep 26 01:05:01 dist-test-slave-l73v krb5kdc[4422](info): commencing operation
krb5kdc: starting...
W20260926 01:05:02.099006 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:05:02 dist-test-slave-l73v krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:05:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:05:02.226838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32829
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:02.311121 4435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:02.311347 4435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:02.311371 4435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:02.312670 4435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:02.312728 4435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:02.312741 4435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:02.312752 4435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:02.312763 4435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:02.314432 4435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:02.314711 4435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:02.314936 4435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:02.320117 4441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:02.320124 4443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:02.320122 4440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:02.320668 4435 server_base.cc:1034] running on GCE node
I20260926 01:05:02.320904 4435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:02.321183 4435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:02.322345 4435 hybrid_clock.cc:648] HybridClock initialized: now 1790384702322310 us; error 51 us; skew 500 ppm
Sep 26 01:05:02 dist-test-slave-l73v krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:02.325255 4435 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:05:02.325589 4435 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:02.325639 4435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:02.325734 4435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:02.326997 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "dddcc6bfb6144b3ca6548724e59cd625"
format_stamp: "Formatted at 2026-09-26 01:05:02 on dist-test-slave-l73v"
server_key: "923ec9b773da47cfe54eb28a798897d3"
server_key_iv: "f39168d3b02bb6386a028030b9185c1c"
server_key_version: "encryptionkey@0"
I20260926 01:05:02.327481 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "dddcc6bfb6144b3ca6548724e59cd625"
format_stamp: "Formatted at 2026-09-26 01:05:02 on dist-test-slave-l73v"
server_key: "923ec9b773da47cfe54eb28a798897d3"
server_key_iv: "f39168d3b02bb6386a028030b9185c1c"
server_key_version: "encryptionkey@0"
I20260926 01:05:02.328831 4435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:02.329653 4449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:02.329819 4435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:02.329901 4435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "dddcc6bfb6144b3ca6548724e59cd625"
format_stamp: "Formatted at 2026-09-26 01:05:02 on dist-test-slave-l73v"
server_key: "923ec9b773da47cfe54eb28a798897d3"
server_key_iv: "f39168d3b02bb6386a028030b9185c1c"
server_key_version: "encryptionkey@0"
I20260926 01:05:02.329963 4435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:02.343088 4435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:02.345283 4435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:02.345402 4435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:02.349938 4435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32829
I20260926 01:05:02.349965 4501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32829 every 8 connection(s)
I20260926 01:05:02.350258 4435 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:02.351238 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:02.351876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 4435
I20260926 01:05:02.351972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:02.352139 392 external_mini_cluster.cc:1428] Setting key b814e39d59f06de5cf6498a053a2bdf9
I20260926 01:05:02.354244 4502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625: Bootstrap starting.
I20260926 01:05:02.355050 4502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:02.355356 4502 log.cc:824] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:02.356292 4502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625: No bootstrap required, opened a new log
I20260926 01:05:02.358726 4502 raft_consensus.cc:348] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddcc6bfb6144b3ca6548724e59cd625" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32829 } }
I20260926 01:05:02.358932 4502 raft_consensus.cc:374] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:02.358973 4502 raft_consensus.cc:729] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dddcc6bfb6144b3ca6548724e59cd625, State: Initialized, Role: FOLLOWER
I20260926 01:05:02.359122 4502 consensus_queue.cc:260] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddcc6bfb6144b3ca6548724e59cd625" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32829 } }
I20260926 01:05:02.359211 4502 raft_consensus.cc:388] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:02.359252 4502 raft_consensus.cc:482] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:02.359302 4502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 01:05:02 dist-test-slave-l73v krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:02.360304 4502 raft_consensus.cc:504] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddcc6bfb6144b3ca6548724e59cd625" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32829 } }
I20260926 01:05:02.360427 4502 leader_election.cc:302] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dddcc6bfb6144b3ca6548724e59cd625; no voters:
I20260926 01:05:02.360621 4502 leader_election.cc:288] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:02.360786 4508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:02.360901 4508 raft_consensus.cc:686] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [term 1 LEADER]: Becoming Leader. State: Replica: dddcc6bfb6144b3ca6548724e59cd625, State: Running, Role: LEADER
I20260926 01:05:02.360917 4502 sys_catalog.cc:564] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:02.361421 4508 consensus_queue.cc:237] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddcc6bfb6144b3ca6548724e59cd625" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32829 } }
I20260926 01:05:02.362591 4507 sys_catalog.cc:455] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dddcc6bfb6144b3ca6548724e59cd625. Latest consensus state: current_term: 1 leader_uuid: "dddcc6bfb6144b3ca6548724e59cd625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddcc6bfb6144b3ca6548724e59cd625" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32829 } } }
I20260926 01:05:02.362679 4507 sys_catalog.cc:458] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:02.362782 4509 sys_catalog.cc:455] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dddcc6bfb6144b3ca6548724e59cd625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddcc6bfb6144b3ca6548724e59cd625" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32829 } } }
I20260926 01:05:02.363152 4509 sys_catalog.cc:458] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625 [sys.catalog]: This master's current role is: LEADER
W20260926 01:05:02.363749 4521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:05:02.363792 4521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:05:02.363948 4524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:02.364226 4524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:02.364616 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:02.352705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.0.98.62:32829)
0926 01:05:02.352908 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:02.352918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:02.352953 (+ 35us) server_negotiation.cc:407] Connection header received
0926 01:05:02.353489 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:02.353498 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:02.353568 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:02.353723 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:02.354283 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:02.355684 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:02.356553 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:02.357186 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:02.357323 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:02.360050 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:02.360056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:02.360056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:02.360071 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:02.362614 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:02.363189 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:02.363192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:02.363193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:02.363312 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:02.363558 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:02.363559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:02.363559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:02.363897 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:02.364039 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:02.364254 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 01:05:02.364399 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260926 01:05:02.365968 4524 catalog_manager.cc:1347] Generated new cluster ID: a76403aa8eab4f46be48c6ba1febc0af
I20260926 01:05:02.366024 4524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:02.377853 4524 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:02.378734 4524 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:02.389732 4524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625: Generated new TSK 0
I20260926 01:05:02.389961 4524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:05:02.427192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32829
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:02.542974 4529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:02.543206 4529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:02.543236 4529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:02.544721 4529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:02.544795 4529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:02.544837 4529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:02.546864 4529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:02.547111 4529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:02.547322 4529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:02.552699 4535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:02.552836 4534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:02.552699 4537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:02.552948 4529 server_base.cc:1034] running on GCE node
I20260926 01:05:02.553182 4529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:02.553449 4529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:02.554620 4529 hybrid_clock.cc:648] HybridClock initialized: now 1790384702554602 us; error 35 us; skew 500 ppm
Sep 26 01:05:02 dist-test-slave-l73v krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:02.562636 4529 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:05:02.563046 4529 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:02.563122 4529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:02.563232 4529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:02.564328 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "08690c8d1a5047189053bf204c5fa702"
format_stamp: "Formatted at 2026-09-26 01:05:02 on dist-test-slave-l73v"
server_key: "bf72688492e37b06b8ff3f72dcb66bf4"
server_key_iv: "8a91f726f80808b25b5819316f6c10fc"
server_key_version: "encryptionkey@0"
I20260926 01:05:02.564634 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08690c8d1a5047189053bf204c5fa702"
format_stamp: "Formatted at 2026-09-26 01:05:02 on dist-test-slave-l73v"
server_key: "bf72688492e37b06b8ff3f72dcb66bf4"
server_key_iv: "8a91f726f80808b25b5819316f6c10fc"
server_key_version: "encryptionkey@0"
I20260926 01:05:02.565922 4529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:02.566833 4543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:02.567022 4529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:05:02.567144 4529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "08690c8d1a5047189053bf204c5fa702"
format_stamp: "Formatted at 2026-09-26 01:05:02 on dist-test-slave-l73v"
server_key: "bf72688492e37b06b8ff3f72dcb66bf4"
server_key_iv: "8a91f726f80808b25b5819316f6c10fc"
server_key_version: "encryptionkey@0"
I20260926 01:05:02.567214 4529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:02.594136 4529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:02.596382 4529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:02.596508 4529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:02.596709 4529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:02.596994 4529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:02.597024 4529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:02.597046 4529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:02.597057 4529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:02.602813 4529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38953
I20260926 01:05:02.602877 4656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38953 every 8 connection(s)
I20260926 01:05:02.603142 4529 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:05:02.603807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 4529
I20260926 01:05:02.603895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:05:02.604094 392 external_mini_cluster.cc:1428] Setting key 955842aeb8c9512c92d51558f69c41de
Sep 26 01:05:02 dist-test-slave-l73v krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:05:02.613736 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:02.604134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36763 (local address 127.0.98.62:32829)
0926 01:05:02.604194 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:02.604196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:02.604383 (+ 187us) server_negotiation.cc:407] Connection header received
0926 01:05:02.605155 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:02.605156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:02.605166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:02.605215 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:02.606521 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:02.607171 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:02.608162 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:02.608468 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:02.608543 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:02.610846 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:02.610848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:02.610848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:02.610858 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:02.612338 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:02.612770 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:02.612771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:02.612772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:02.612825 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:02.613048 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:02.613049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:02.613049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:02.613172 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:02.613227 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:02.613539 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 01:05:02.613640 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:05:02.613924 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:02.604104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32829 (local address 127.0.98.1:36763)
0926 01:05:02.604260 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:02.604304 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:02.604994 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:02.605311 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:02.605316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:02.605708 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:02.606427 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:02.606435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:02.607266 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:02.607267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:02.608085 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:02.608087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:02.608598 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:02.608599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:02.608720 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:02.609303 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:02.609311 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:02.610742 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:02.612431 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:02.612441 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:02.612444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:02.612696 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:02.612882 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:02.612882 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:02.612883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:02.612991 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:02.613227 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:02.613233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:02.613410 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 01:05:02.613512 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 01:05:02.613694 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 01:05:02.614437 4657 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32829
I20260926 01:05:02.614548 4657 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:02.614739 4657 heartbeater.cc:511] Master 127.0.98.62:32829 requested a full tablet report, sending...
I20260926 01:05:02.615446 4459 ts_manager.cc:194] Registered new tserver with Master: 08690c8d1a5047189053bf204c5fa702 (127.0.98.1:38953)
I20260926 01:05:02.616137 4459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260926 01:05:02.737963 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.133s user 0.003s sys 0.000s
I20260926 01:05:02.738360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32829
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:02.832748 4662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:02.832994 4662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:02.833021 4662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:02.834439 4662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:02.834507 4662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:02.834540 4662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:02.836542 4662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:02.837045 4662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:02.837370 4662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:02.843204 4668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:02.843205 4667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:02.843238 4662 server_base.cc:1034] running on GCE node
W20260926 01:05:02.843205 4670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:02.843757 4662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:02.844050 4662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:02.845230 4662 hybrid_clock.cc:648] HybridClock initialized: now 1790384702845197 us; error 47 us; skew 500 ppm
Sep 26 01:05:02 dist-test-slave-l73v krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:02.848063 4662 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:05:02.848438 4662 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:02.848495 4662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:02.848628 4662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:02.849728 4662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "1208724841894a68a01862875d05cfca"
format_stamp: "Formatted at 2026-09-26 01:05:02 on dist-test-slave-l73v"
server_key: "609236258cded7f3269a5fb31d70fc88"
server_key_iv: "b4f63b4c1d36318ef1225933cd1d8097"
server_key_version: "encryptionkey@0"
I20260926 01:05:02.850086 4662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1208724841894a68a01862875d05cfca"
format_stamp: "Formatted at 2026-09-26 01:05:02 on dist-test-slave-l73v"
server_key: "609236258cded7f3269a5fb31d70fc88"
server_key_iv: "b4f63b4c1d36318ef1225933cd1d8097"
server_key_version: "encryptionkey@0"
I20260926 01:05:02.851619 4662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:02.852738 4676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:02.853024 4662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:02.853127 4662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "1208724841894a68a01862875d05cfca"
format_stamp: "Formatted at 2026-09-26 01:05:02 on dist-test-slave-l73v"
server_key: "609236258cded7f3269a5fb31d70fc88"
server_key_iv: "b4f63b4c1d36318ef1225933cd1d8097"
server_key_version: "encryptionkey@0"
I20260926 01:05:02.853210 4662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:02.869156 4662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:02.871783 4662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:02.871944 4662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:02.872203 4662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:02.872663 4662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:02.872735 4662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:02.872771 4662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:02.872787 4662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:02.880173 4662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46153
I20260926 01:05:02.880237 4789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46153 every 8 connection(s)
I20260926 01:05:02.880501 4662 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:05:02.883890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 4662
I20260926 01:05:02.883986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:05:02.884140 392 external_mini_cluster.cc:1428] Setting key 4ab81c0fa6f4fdd90cb07599375ad6a2
Sep 26 01:05:02 dist-test-slave-l73v krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:02.891072 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:02.881319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50389 (local address 127.0.98.62:32829)
0926 01:05:02.881374 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:02.881376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:02.881538 (+ 162us) server_negotiation.cc:407] Connection header received
0926 01:05:02.881990 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:02.881991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:02.882003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:02.882061 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:02.883108 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:02.883617 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:02.885099 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:02.885407 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:02.885481 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:02.888089 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:02.888091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:02.888091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:02.888099 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:02.889515 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:02.890057 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:02.890059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:02.890059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:02.890116 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:02.890402 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:02.890403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:02.890404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:02.890527 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:02.890585 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:02.890882 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 01:05:02.890975 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:05:02.891265 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:02.881177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32829 (local address 127.0.98.2:50389)
0926 01:05:02.881388 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:02.881430 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:02.881912 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:02.882131 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:02.882135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:02.882455 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:02.882985 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:02.882992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:02.883741 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:02.883742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:02.884964 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:02.884974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:02.885519 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:02.885520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:02.885663 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:02.886210 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:02.886221 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:02.887972 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:02.889612 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:02.889625 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:02.889628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:02.889983 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:02.890182 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:02.890183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:02.890183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:02.890326 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:02.890587 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:02.890594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:02.890754 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 01:05:02.890850 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 01:05:02.891028 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260926 01:05:02.891692 4790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32829
I20260926 01:05:02.891790 4790 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:05:02.891948 4790 heartbeater.cc:511] Master 127.0.98.62:32829 requested a full tablet report, sending...
I20260926 01:05:02.892350 4459 ts_manager.cc:194] Registered new tserver with Master: 1208724841894a68a01862875d05cfca (127.0.98.2:46153)
I20260926 01:05:02.892707 4459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260926 01:05:03.077028 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.192s user 0.000s sys 0.003s
I20260926 01:05:03.077423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32829
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:03.163437 4795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:03.163673 4795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:03.163702 4795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:03.164997 4795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:03.165074 4795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:03.165108 4795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:03.166868 4795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:03.167156 4795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:03.167413 4795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:05:03.173079 4795 server_base.cc:1034] running on GCE node
W20260926 01:05:03.173062 4801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:03.173038 4800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:03.173026 4803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:03.173517 4795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:03.173770 4795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:03.174935 4795 hybrid_clock.cc:648] HybridClock initialized: now 1790384703174901 us; error 36 us; skew 500 ppm
Sep 26 01:05:03 dist-test-slave-l73v krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:03.177663 4795 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:05:03.178082 4795 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:03.178144 4795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:03.178290 4795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:03.179504 4795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "abcf6ef7c44f49f3b0eadf9751649b85"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "4fc978911e7831d40386da3194f07b6d"
server_key_iv: "15cd1620fc9f8ea07b1fde15860d3b0f"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.179891 4795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "abcf6ef7c44f49f3b0eadf9751649b85"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "4fc978911e7831d40386da3194f07b6d"
server_key_iv: "15cd1620fc9f8ea07b1fde15860d3b0f"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.181363 4795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:03.182253 4809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:03.182495 4795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:03.182598 4795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "abcf6ef7c44f49f3b0eadf9751649b85"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "4fc978911e7831d40386da3194f07b6d"
server_key_iv: "15cd1620fc9f8ea07b1fde15860d3b0f"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.182741 4795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:03.202278 4795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:03.204480 4795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:03.204631 4795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:03.204869 4795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:03.205200 4795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:03.205235 4795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:03.205263 4795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:03.205281 4795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:03.210819 4795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34905
I20260926 01:05:03.210857 4922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34905 every 8 connection(s)
I20260926 01:05:03.211124 4795 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:05:03.212461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 4795
I20260926 01:05:03.212563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:05:03.212774 392 external_mini_cluster.cc:1428] Setting key 65e352bb34521bfe29acf01bbeda5147
Sep 26 01:05:03 dist-test-slave-l73v krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:03.221423 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.211841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35303 (local address 127.0.98.62:32829)
0926 01:05:03.211917 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.211920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.212279 (+ 359us) server_negotiation.cc:407] Connection header received
0926 01:05:03.212896 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.212897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.212910 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.212970 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:03.214063 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.214624 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.215717 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.215919 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.215977 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.218362 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:03.218364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:03.218364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:03.218371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:03.219749 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:03.220262 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:03.220264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:03.220264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:03.220330 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:03.220667 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:03.220668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:03.220669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:03.220807 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:03.220864 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.221239 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.221328 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 01:05:03.221648 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.211814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32829 (local address 127.0.98.3:35303)
0926 01:05:03.212087 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:03.212159 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:03.212771 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:03.213060 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:03.213065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:03.213420 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:03.213962 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.213968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.214739 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.214740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.215621 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.215624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.216014 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.216015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.216152 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.216694 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:03.216704 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:03.218257 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:03.219824 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:03.219833 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:03.219835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:03.220138 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:03.220406 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:03.220407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:03.220407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:03.220554 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:03.220885 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:03.220892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:03.221069 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 01:05:03.221183 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 01:05:03.221412 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":80,"threads_started":1}
I20260926 01:05:03.222165 4923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32829
I20260926 01:05:03.222293 4923 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:03.222481 4923 heartbeater.cc:511] Master 127.0.98.62:32829 requested a full tablet report, sending...
I20260926 01:05:03.222905 4459 ts_manager.cc:194] Registered new tserver with Master: abcf6ef7c44f49f3b0eadf9751649b85 (127.0.98.3:34905)
I20260926 01:05:03.223274 4459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35303
I20260926 01:05:03.224380 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:05:03.231896 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.225646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.0.98.62:32829)
0926 01:05:03.225768 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.225770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.225823 (+ 53us) server_negotiation.cc:407] Connection header received
0926 01:05:03.225898 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.225899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.225908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.225953 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:03.226584 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.227086 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.227813 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.228008 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.228061 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.229229 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:03.229231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:03.229231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:03.229239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:03.230655 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:03.231006 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:03.231007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:03.231008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:03.231114 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:03.231303 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:03.231304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:03.231304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:03.231475 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:03.231575 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.231704 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.231797 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:05:03.233770 4459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:05:03.235504 4459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:05:03.242107 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.238421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.0.98.2:46153)
0926 01:05:03.238765 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.238770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.238780 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:03.238817 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.238819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.238833 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.238928 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:03.239040 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.239843 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.240988 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.241670 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.241750 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.241816 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.241966 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.242013 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":95,"threads_started":1}
I20260926 01:05:03.242189 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.238287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46153 (local address 127.0.0.1:48892)
0926 01:05:03.238414 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:03.238422 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:03.238461 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:03.238928 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:03.238929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:03.238931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:03.238987 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.238988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.239944 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.239945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.240914 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.240917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.241781 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.241782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.241850 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.241863 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 01:05:03.241950 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 01:05:03.242013 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":32,"thread_start_us":65,"threads_started":1}
I20260926 01:05:03.242503 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.237829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38953 (local address 127.0.0.1:36200)
0926 01:05:03.237965 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:03.237997 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:03.238068 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:03.238328 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:03.238331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:03.238334 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:03.238429 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.238432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.239746 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.239747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.241131 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.241134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.241974 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.241975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.242028 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.242032 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:03.242082 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:05:03.242443 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260926 01:05:03.242835 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.237944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.0.98.1:38953)
0926 01:05:03.238109 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.238114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.238125 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:03.238159 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.238161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.238177 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.238246 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:03.238518 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.239657 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.241217 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.241883 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.241945 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.242000 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.242147 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.242757 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":56,"threads_started":1}
I20260926 01:05:03.243144 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.238097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34905 (local address 127.0.0.1:55546)
0926 01:05:03.238234 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:03.238244 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:03.238292 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:03.238835 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:03.238836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:03.238838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:03.238878 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.238880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.240052 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.240053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.241132 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.241134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.242924 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.242925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.242991 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.242995 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:03.243050 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:05:03.243098 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260926 01:05:03.243011 4724 tablet_service.cc:1467] Processing CreateTablet for tablet fd0955811aad4ec6bce6b91c3db0b171 (DEFAULT_TABLE table=test-table [id=07890718b3744ede830b2182ec06d2ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:03.243394 4724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0955811aad4ec6bce6b91c3db0b171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:03.243417 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.238512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.0.98.3:34905)
0926 01:05:03.238671 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.238676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.238685 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:03.238718 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.238720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.238734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.238940 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:03.239008 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.239963 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.242051 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.242792 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.242867 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.242922 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.243302 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.243350 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260926 01:05:03.243604 4591 tablet_service.cc:1467] Processing CreateTablet for tablet fd0955811aad4ec6bce6b91c3db0b171 (DEFAULT_TABLE table=test-table [id=07890718b3744ede830b2182ec06d2ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:03.243906 4591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0955811aad4ec6bce6b91c3db0b171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:03.244001 4857 tablet_service.cc:1467] Processing CreateTablet for tablet fd0955811aad4ec6bce6b91c3db0b171 (DEFAULT_TABLE table=test-table [id=07890718b3744ede830b2182ec06d2ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:03.244323 4857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0955811aad4ec6bce6b91c3db0b171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:03.246850 4940 tablet_bootstrap.cc:492] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: Bootstrap starting.
I20260926 01:05:03.247244 4942 tablet_bootstrap.cc:492] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: Bootstrap starting.
I20260926 01:05:03.247294 4941 tablet_bootstrap.cc:492] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702: Bootstrap starting.
I20260926 01:05:03.247637 4940 tablet_bootstrap.cc:654] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:03.247843 4942 tablet_bootstrap.cc:654] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:03.247884 4941 tablet_bootstrap.cc:654] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:03.248138 4941 log.cc:824] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:03.248263 4940 log.cc:824] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:03.248395 4942 log.cc:824] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:03.248878 4941 tablet_bootstrap.cc:492] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702: No bootstrap required, opened a new log
I20260926 01:05:03.248941 4941 ts_tablet_manager.cc:1397] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:03.249162 4942 tablet_bootstrap.cc:492] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: No bootstrap required, opened a new log
I20260926 01:05:03.249217 4942 ts_tablet_manager.cc:1397] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:03.250730 4942 raft_consensus.cc:348] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 } }
I20260926 01:05:03.250917 4942 raft_consensus.cc:374] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:03.250958 4942 raft_consensus.cc:729] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1208724841894a68a01862875d05cfca, State: Initialized, Role: FOLLOWER
I20260926 01:05:03.251039 4940 tablet_bootstrap.cc:492] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: No bootstrap required, opened a new log
I20260926 01:05:03.251098 4940 ts_tablet_manager.cc:1397] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 01:05:03.251109 4942 consensus_queue.cc:260] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 } }
I20260926 01:05:03.251049 4941 raft_consensus.cc:348] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 } }
I20260926 01:05:03.251195 4941 raft_consensus.cc:374] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:03.251230 4941 raft_consensus.cc:729] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08690c8d1a5047189053bf204c5fa702, State: Initialized, Role: FOLLOWER
I20260926 01:05:03.251330 4941 consensus_queue.cc:260] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 } }
I20260926 01:05:03.251384 4942 ts_tablet_manager.cc:1428] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:05:03.251525 4790 heartbeater.cc:503] Master 127.0.98.62:32829 was elected leader, sending a full tablet report...
I20260926 01:05:03.251578 4941 ts_tablet_manager.cc:1428] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:05:03.251713 4657 heartbeater.cc:503] Master 127.0.98.62:32829 was elected leader, sending a full tablet report...
I20260926 01:05:03.252964 4940 raft_consensus.cc:348] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 } }
I20260926 01:05:03.253091 4940 raft_consensus.cc:374] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:03.253111 4940 raft_consensus.cc:729] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcf6ef7c44f49f3b0eadf9751649b85, State: Initialized, Role: FOLLOWER
I20260926 01:05:03.253202 4940 consensus_queue.cc:260] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 } }
I20260926 01:05:03.253449 4940 ts_tablet_manager.cc:1428] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:05:03.253486 4923 heartbeater.cc:503] Master 127.0.98.62:32829 was elected leader, sending a full tablet report...
W20260926 01:05:03.354585 4658 tablet.cc:2369] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:05:03.381253 4791 tablet.cc:2369] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:05:03.428094 4948 raft_consensus.cc:482] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:03.428253 4948 raft_consensus.cc:504] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 } }
I20260926 01:05:03.428618 4948 leader_election.cc:288] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08690c8d1a5047189053bf204c5fa702 (127.0.98.1:38953), 1208724841894a68a01862875d05cfca (127.0.98.2:46153)
I20260926 01:05:03.431573 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.428921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38597 (local address 127.0.98.1:38953)
0926 01:05:03.428980 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.428982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.428995 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:05:03.429024 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.429025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.429037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.429094 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:03.429380 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.429856 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.430868 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.431250 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.431314 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.431326 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.431461 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.431499 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:05:03.431573 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.428817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38953 (local address 127.0.98.3:38597)
0926 01:05:03.428875 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:03.428894 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:03.428956 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:03.429205 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:03.429206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:03.429208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:03.429271 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.429273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.429931 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.429932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.430778 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.430781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.431325 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.431325 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.431397 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.431400 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:03.431445 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:05:03.431491 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 01:05:03.431704 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.429004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37131 (local address 127.0.98.2:46153)
0926 01:05:03.429043 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.429044 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.429159 (+ 115us) server_negotiation.cc:407] Connection header received
0926 01:05:03.429208 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.429209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.429219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.429266 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:03.429402 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.429975 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.430990 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.431359 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.431424 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.431435 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.431588 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.431638 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:05:03.431867 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.428922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46153 (local address 127.0.98.3:37131)
0926 01:05:03.429082 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:03.429099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:03.429151 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:03.429272 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:03.429273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:03.429275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:03.429342 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.429344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.430074 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.430075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.430894 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:03.430896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.431437 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.431438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:03.431520 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.431523 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:03.431568 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:05:03.431608 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260926 01:05:03.431968 4611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0955811aad4ec6bce6b91c3db0b171" candidate_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08690c8d1a5047189053bf204c5fa702" is_pre_election: true
I20260926 01:05:03.432154 4611 raft_consensus.cc:2454] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abcf6ef7c44f49f3b0eadf9751649b85 in term 0.
I20260926 01:05:03.432391 4812 leader_election.cc:302] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08690c8d1a5047189053bf204c5fa702, abcf6ef7c44f49f3b0eadf9751649b85; no voters:
I20260926 01:05:03.432523 4948 raft_consensus.cc:2781] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:05:03.432632 4948 raft_consensus.cc:482] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:03.432662 4948 raft_consensus.cc:3037] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:03.432732 4744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0955811aad4ec6bce6b91c3db0b171" candidate_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1208724841894a68a01862875d05cfca" is_pre_election: true
I20260926 01:05:03.432845 4744 raft_consensus.cc:2454] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abcf6ef7c44f49f3b0eadf9751649b85 in term 0.
I20260926 01:05:03.433501 4948 raft_consensus.cc:504] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 } }
I20260926 01:05:03.433624 4948 leader_election.cc:288] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [CANDIDATE]: Term 1 election: Requested vote from peers 08690c8d1a5047189053bf204c5fa702 (127.0.98.1:38953), 1208724841894a68a01862875d05cfca (127.0.98.2:46153)
I20260926 01:05:03.433830 4611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0955811aad4ec6bce6b91c3db0b171" candidate_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08690c8d1a5047189053bf204c5fa702"
I20260926 01:05:03.433861 4744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0955811aad4ec6bce6b91c3db0b171" candidate_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1208724841894a68a01862875d05cfca"
I20260926 01:05:03.433928 4744 raft_consensus.cc:3037] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:03.433923 4611 raft_consensus.cc:3037] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:03.434873 4611 raft_consensus.cc:2454] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abcf6ef7c44f49f3b0eadf9751649b85 in term 1.
I20260926 01:05:03.434873 4744 raft_consensus.cc:2454] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abcf6ef7c44f49f3b0eadf9751649b85 in term 1.
I20260926 01:05:03.435092 4812 leader_election.cc:302] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1208724841894a68a01862875d05cfca, abcf6ef7c44f49f3b0eadf9751649b85; no voters:
I20260926 01:05:03.435189 4948 raft_consensus.cc:2781] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:03.435364 4948 raft_consensus.cc:686] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 1 LEADER]: Becoming Leader. State: Replica: abcf6ef7c44f49f3b0eadf9751649b85, State: Running, Role: LEADER
I20260926 01:05:03.435462 4948 consensus_queue.cc:237] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } } peers { permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 } }
I20260926 01:05:03.436362 4459 catalog_manager.cc:5495] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to abcf6ef7c44f49f3b0eadf9751649b85 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abcf6ef7c44f49f3b0eadf9751649b85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 } health_report { overall_health: UNKNOWN } } }
W20260926 01:05:03.461735 4924 tablet.cc:2369] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:05:03.474475 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.472243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.0.98.3:34905)
0926 01:05:03.472300 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.472303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.472311 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:03.472363 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.472364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.472383 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.472440 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:03.472554 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.473015 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.473776 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:03.473972 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:03.474033 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:03.474156 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:03.474261 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:03.474319 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.474349 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.474368 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:05:03.476312 4744 raft_consensus.cc:1261] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca [term 1 FOLLOWER]: Refusing update from remote peer abcf6ef7c44f49f3b0eadf9751649b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:03.476315 4611 raft_consensus.cc:1261] T fd0955811aad4ec6bce6b91c3db0b171 P 08690c8d1a5047189053bf204c5fa702 [term 1 FOLLOWER]: Refusing update from remote peer abcf6ef7c44f49f3b0eadf9751649b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:03.476660 4948 consensus_queue.cc:1035] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08690c8d1a5047189053bf204c5fa702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:03.476804 4950 consensus_queue.cc:1035] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1208724841894a68a01862875d05cfca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:03.479768 4954 mvcc.cc:205] Tried to move back new op lower bound from 7333415745436155904 to 7333415745272213504. Current Snapshot: MvccSnapshot[applied={T|T < 7333415745436155904 or (T in {7333415745436155904})}]
I20260926 01:05:03.481535 4466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:03.481667 4466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:03.482517 4466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dddcc6bfb6144b3ca6548724e59cd625: Sending DeleteTablet for 3 replicas of tablet fd0955811aad4ec6bce6b91c3db0b171
I20260926 01:05:03.483052 4591 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0955811aad4ec6bce6b91c3db0b171 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36200
I20260926 01:05:03.483105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 4529
I20260926 01:05:03.483572 4857 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0955811aad4ec6bce6b91c3db0b171 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55546
I20260926 01:05:03.483868 4963 tablet_replica.cc:330] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: stopping tablet replica
I20260926 01:05:03.483973 4963 raft_consensus.cc:2229] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:05:03.484016 4724 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0955811aad4ec6bce6b91c3db0b171 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48892
I20260926 01:05:03.484067 4963 raft_consensus.cc:2258] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:03.484234 4964 tablet_replica.cc:330] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: stopping tablet replica
I20260926 01:05:03.484329 4964 raft_consensus.cc:2229] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:03.484443 4964 raft_consensus.cc:2258] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:03.484952 4964 ts_tablet_manager.cc:1905] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:03.484962 4963 ts_tablet_manager.cc:1905] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:03.486778 4963 ts_tablet_manager.cc:1918] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:03.486799 4964 ts_tablet_manager.cc:1918] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:03.486856 4963 log.cc:1196] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/fd0955811aad4ec6bce6b91c3db0b171
I20260926 01:05:03.486866 4964 log.cc:1196] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/fd0955811aad4ec6bce6b91c3db0b171
I20260926 01:05:03.487133 4964 ts_tablet_manager.cc:1939] T fd0955811aad4ec6bce6b91c3db0b171 P 1208724841894a68a01862875d05cfca: Deleting consensus metadata
I20260926 01:05:03.487143 4963 ts_tablet_manager.cc:1939] T fd0955811aad4ec6bce6b91c3db0b171 P abcf6ef7c44f49f3b0eadf9751649b85: Deleting consensus metadata
I20260926 01:05:03.487564 4452 catalog_manager.cc:4841] TS abcf6ef7c44f49f3b0eadf9751649b85 (127.0.98.3:34905): tablet fd0955811aad4ec6bce6b91c3db0b171 (table test-table [id=07890718b3744ede830b2182ec06d2ce]) successfully deleted
I20260926 01:05:03.487675 4452 catalog_manager.cc:4841] TS 1208724841894a68a01862875d05cfca (127.0.98.2:46153): tablet fd0955811aad4ec6bce6b91c3db0b171 (table test-table [id=07890718b3744ede830b2182ec06d2ce]) successfully deleted
W20260926 01:05:03.490129 4452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38953 (error 108)
I20260926 01:05:03.490424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 4662
I20260926 01:05:03.491170 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.490559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38953 (local address 127.0.0.1:36204)
0926 01:05:03.491072 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:03.491131 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488}
W20260926 01:05:03.491336 4452 catalog_manager.cc:4584] TS 08690c8d1a5047189053bf204c5fa702 (127.0.98.1:38953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd0955811aad4ec6bce6b91c3db0b171: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38953: connect: Connection refused (error 111)
I20260926 01:05:03.495285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 4795
I20260926 01:05:03.499797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 4435
2026-09-26T01:05:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2369 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T01:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:03Z Disabled control of system clock
I20260926 01:05:03.526777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43963
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 01:05:03.608430 4969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:03.608691 4969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:03.608716 4969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:03.609947 4969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:03.609992 4969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:03.610003 4969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:03.610015 4969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:03.610026 4969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:03.611531 4969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:03.611775 4969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:03.612040 4969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:03.617367 4977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:03.617393 4975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:03.617412 4969 server_base.cc:1034] running on GCE node
W20260926 01:05:03.617367 4974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:03.618034 4969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:03.618402 4969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:03.619611 4969 hybrid_clock.cc:648] HybridClock initialized: now 1790384703619541 us; error 62 us; skew 500 ppm
I20260926 01:05:03.620986 4969 webserver.cc:460] Webserver started at http://127.0.98.62:45633/ using document root <none> and password file <none>
I20260926 01:05:03.621230 4969 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:03.621290 4969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:03.621433 4969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:03.622606 4969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "7d78e0ddf9e0462dae6c7049f777ed32"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "1886fe1861b6590d0076d381d8426345"
server_key_iv: "9160d321cf12ef7c23646c43d80ccf6e"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.622949 4969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d78e0ddf9e0462dae6c7049f777ed32"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "1886fe1861b6590d0076d381d8426345"
server_key_iv: "9160d321cf12ef7c23646c43d80ccf6e"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.624243 4969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:03.625208 4983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:03.625452 4969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:03.625548 4969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "7d78e0ddf9e0462dae6c7049f777ed32"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "1886fe1861b6590d0076d381d8426345"
server_key_iv: "9160d321cf12ef7c23646c43d80ccf6e"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.625617 4969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:03.629235 4969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:03.629519 4969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:03.629643 4969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:03.633440 4969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43963
I20260926 01:05:03.633484 5035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43963 every 8 connection(s)
I20260926 01:05:03.633841 4969 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:03.634608 5036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:03.637467 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32: Bootstrap starting.
I20260926 01:05:03.638026 5036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:03.638278 5036 log.cc:824] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:03.639056 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32: No bootstrap required, opened a new log
I20260926 01:05:03.640740 5036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d78e0ddf9e0462dae6c7049f777ed32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43963 } }
I20260926 01:05:03.640899 5036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:03.640920 5036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d78e0ddf9e0462dae6c7049f777ed32, State: Initialized, Role: FOLLOWER
I20260926 01:05:03.641058 5036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d78e0ddf9e0462dae6c7049f777ed32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43963 } }
I20260926 01:05:03.641129 5036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:03.641153 5036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:03.641208 5036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:03.641417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 4969
I20260926 01:05:03.641496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:03.641657 392 external_mini_cluster.cc:1428] Setting key 32acd4324b9c73272a5cf9abf268496f
I20260926 01:05:03.641850 5036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d78e0ddf9e0462dae6c7049f777ed32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43963 } }
I20260926 01:05:03.641965 5036 leader_election.cc:302] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d78e0ddf9e0462dae6c7049f777ed32; no voters:
I20260926 01:05:03.642156 5036 leader_election.cc:288] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:03.642234 5040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:03.642364 5040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [term 1 LEADER]: Becoming Leader. State: Replica: 7d78e0ddf9e0462dae6c7049f777ed32, State: Running, Role: LEADER
I20260926 01:05:03.642467 5036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:03.642473 5040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d78e0ddf9e0462dae6c7049f777ed32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43963 } }
I20260926 01:05:03.642936 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d78e0ddf9e0462dae6c7049f777ed32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d78e0ddf9e0462dae6c7049f777ed32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43963 } } }
I20260926 01:05:03.643023 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:03.643137 5043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d78e0ddf9e0462dae6c7049f777ed32. Latest consensus state: current_term: 1 leader_uuid: "7d78e0ddf9e0462dae6c7049f777ed32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d78e0ddf9e0462dae6c7049f777ed32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43963 } } }
I20260926 01:05:03.643211 5043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32 [sys.catalog]: This master's current role is: LEADER
W20260926 01:05:03.644254 5056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:05:03.644318 5056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:05:03.644462 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.642227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.0.98.62:43963)
0926 01:05:03.642800 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.642809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.642842 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:05:03.642966 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.642974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.643021 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.643419 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:03.644024 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:03.644027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:03.644027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:03.644030 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:03.644057 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:03.644067 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:03.644122 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.644217 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.644280 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":100,"threads_started":1}
I20260926 01:05:03.646270 5050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:03.646528 5050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:03.648506 5050 catalog_manager.cc:1347] Generated new cluster ID: 38f3f3f8a77a4e7bac3f8482cad712a9
I20260926 01:05:03.648568 5050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:03.687425 5050 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:03.687621 5050 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:03.692080 5050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32: Generated new TSK 0
I20260926 01:05:03.692286 5050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:05:03.697088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43963
--builtin_ntp_servers=127.0.98.20:35327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 01:05:03.786306 5060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:03.786594 5060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:03.786623 5060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:03.787986 5060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:03.788043 5060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:03.788074 5060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:03.790104 5060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:03.790450 5060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:03.790861 5060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:03.796247 5065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:03.796275 5068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:03.796285 5066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:03.797278 5060 server_base.cc:1034] running on GCE node
I20260926 01:05:03.797622 5060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:03.797981 5060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:03.799237 5060 hybrid_clock.cc:648] HybridClock initialized: now 1790384703799218 us; error 37 us; skew 500 ppm
I20260926 01:05:03.801187 5060 webserver.cc:460] Webserver started at http://127.0.98.1:33825/ using document root <none> and password file <none>
I20260926 01:05:03.801486 5060 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:03.801553 5060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:03.801714 5060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:03.802960 5060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "032b5410fd74415c818ca00da29526d0"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "f41d67b6db3b522e82f6f30838561fef"
server_key_iv: "b06c6c1a178a6d3c236fade4c40d8314"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.803315 5060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "032b5410fd74415c818ca00da29526d0"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "f41d67b6db3b522e82f6f30838561fef"
server_key_iv: "b06c6c1a178a6d3c236fade4c40d8314"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.804723 5060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:03.805608 5074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:03.805823 5060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:03.805922 5060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "032b5410fd74415c818ca00da29526d0"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "f41d67b6db3b522e82f6f30838561fef"
server_key_iv: "b06c6c1a178a6d3c236fade4c40d8314"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.805990 5060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:03.809468 5060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:03.809753 5060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:03.809882 5060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:03.810073 5060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:03.810410 5060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:03.810451 5060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:03.810487 5060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:03.810510 5060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:03.816125 5060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42779
I20260926 01:05:03.816186 5187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42779 every 8 connection(s)
I20260926 01:05:03.816509 5060 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:05:03.820047 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.817224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43637 (local address 127.0.98.62:43963)
0926 01:05:03.817278 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.817280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.817541 (+ 261us) server_negotiation.cc:407] Connection header received
0926 01:05:03.818519 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.818520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.818529 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.818585 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:03.819824 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:03.819825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:03.819826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:03.819831 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:03.819848 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:03.819853 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:03.819913 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.819974 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.819988 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:05:03.820209 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.817211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43963 (local address 127.0.98.1:43637)
0926 01:05:03.817390 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:03.817426 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:03.818376 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:03.818659 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:03.818663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:03.818673 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:03.819644 (+ 971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:03.819644 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:03.819676 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:03.819708 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:03.819915 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:03.819918 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:03.819921 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:03.819975 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 01:05:03.820029 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260926 01:05:03.820791 5188 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43963
I20260926 01:05:03.820879 5188 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:03.821022 5188 heartbeater.cc:511] Master 127.0.98.62:43963 requested a full tablet report, sending...
I20260926 01:05:03.821574 5000 ts_manager.cc:194] Registered new tserver with Master: 032b5410fd74415c818ca00da29526d0 (127.0.98.1:42779)
I20260926 01:05:03.822818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 5060
I20260926 01:05:03.822913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:05:03.823103 392 external_mini_cluster.cc:1428] Setting key de374d9cf1117804a8dcd922127c35c5
I20260926 01:05:03.824261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43963
--builtin_ntp_servers=127.0.98.20:35327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 01:05:03.910713 5191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:03.910945 5191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:03.910977 5191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:03.912250 5191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:03.912297 5191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:03.912335 5191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:03.913787 5191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:03.914053 5191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:03.914358 5191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:03.919500 5197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:03.919628 5199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:03.919745 5191 server_base.cc:1034] running on GCE node
W20260926 01:05:03.919494 5196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:03.920045 5191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:03.920274 5191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:03.921411 5191 hybrid_clock.cc:648] HybridClock initialized: now 1790384703921389 us; error 38 us; skew 500 ppm
I20260926 01:05:03.922706 5191 webserver.cc:460] Webserver started at http://127.0.98.2:40283/ using document root <none> and password file <none>
I20260926 01:05:03.922935 5191 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:03.922979 5191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:03.923106 5191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:03.924401 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "b11787f4e7de450dba0ea190bd20af49"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "bdd97c3f974f108370bd21b76b1ffe99"
server_key_iv: "399dabd3ad380d5a59d54bdfad81a4a2"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.924798 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b11787f4e7de450dba0ea190bd20af49"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "bdd97c3f974f108370bd21b76b1ffe99"
server_key_iv: "399dabd3ad380d5a59d54bdfad81a4a2"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.926450 5191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:05:03.927340 5205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:03.927574 5191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:03.927670 5191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "b11787f4e7de450dba0ea190bd20af49"
format_stamp: "Formatted at 2026-09-26 01:05:03 on dist-test-slave-l73v"
server_key: "bdd97c3f974f108370bd21b76b1ffe99"
server_key_iv: "399dabd3ad380d5a59d54bdfad81a4a2"
server_key_version: "encryptionkey@0"
I20260926 01:05:03.927740 5191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:03.931412 5191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:03.931685 5191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:03.931813 5191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:03.931998 5191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:03.932327 5191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:03.932381 5191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:03.932410 5191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:03.932426 5191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:03.938176 5191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36395
I20260926 01:05:03.938226 5318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36395 every 8 connection(s)
I20260926 01:05:03.938561 5191 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:05:03.939761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 5191
I20260926 01:05:03.939854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:05:03.940017 392 external_mini_cluster.cc:1428] Setting key 97f35615bd653aa95a970b9d4135d4b3
I20260926 01:05:03.940938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43963
--builtin_ntp_servers=127.0.98.20:35327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 01:05:03.941695 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.939315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49059 (local address 127.0.98.62:43963)
0926 01:05:03.939369 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:03.939371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:03.939568 (+ 197us) server_negotiation.cc:407] Connection header received
0926 01:05:03.940168 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:03.940169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:03.940175 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:03.940263 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:03.941461 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:03.941462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:03.941462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:03.941468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:03.941488 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:03.941493 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:03.941557 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:03.941612 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 01:05:03.941626 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:05:03.941954 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:03.939272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43963 (local address 127.0.98.2:49059)
0926 01:05:03.939448 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:03.939483 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:03.940021 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:03.940337 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:03.940340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:03.940350 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:03.941307 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:03.941308 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:03.941340 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:03.941370 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:03.941557 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:03.941563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:03.941566 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:03.941618 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 01:05:03.941669 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260926 01:05:03.942508 5319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43963
I20260926 01:05:03.942600 5319 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:03.942752 5319 heartbeater.cc:511] Master 127.0.98.62:43963 requested a full tablet report, sending...
I20260926 01:05:03.943161 5000 ts_manager.cc:194] Registered new tserver with Master: b11787f4e7de450dba0ea190bd20af49 (127.0.98.2:36395)
W20260926 01:05:04.023089 5322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:04.023335 5322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:04.023367 5322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:04.024953 5322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:04.025020 5322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:04.025063 5322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:04.026726 5322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:04.027141 5322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:04.027611 5322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:04.032933 5330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:04.033025 5328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:04.033138 5322 server_base.cc:1034] running on GCE node
W20260926 01:05:04.032932 5327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:04.033402 5322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:04.033648 5322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:04.034940 5322 hybrid_clock.cc:648] HybridClock initialized: now 1790384704034927 us; error 30 us; skew 500 ppm
I20260926 01:05:04.036448 5322 webserver.cc:460] Webserver started at http://127.0.98.3:43185/ using document root <none> and password file <none>
I20260926 01:05:04.036693 5322 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:04.036746 5322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:04.036877 5322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:04.037976 5322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "02bc4f2c0e514894957c977c6f04f5c8"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "63b722262dab1c58720e2d9cf2480091"
server_key_iv: "03c0cf124053b5a8b0af45c4d0e3c4d2"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.038337 5322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02bc4f2c0e514894957c977c6f04f5c8"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "63b722262dab1c58720e2d9cf2480091"
server_key_iv: "03c0cf124053b5a8b0af45c4d0e3c4d2"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.039683 5322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:04.040573 5336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:04.040867 5322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:04.040973 5322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "02bc4f2c0e514894957c977c6f04f5c8"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "63b722262dab1c58720e2d9cf2480091"
server_key_iv: "03c0cf124053b5a8b0af45c4d0e3c4d2"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.041047 5322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:04.044643 5322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:04.044931 5322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:04.045048 5322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:04.045238 5322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:04.045542 5322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:04.045574 5322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:04.045603 5322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:04.045619 5322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:04.051602 5322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45621
I20260926 01:05:04.051643 5449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45621 every 8 connection(s)
I20260926 01:05:04.052006 5322 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:05:04.055063 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.052773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59653 (local address 127.0.98.62:43963)
0926 01:05:04.052834 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.052836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.053063 (+ 227us) server_negotiation.cc:407] Connection header received
0926 01:05:04.053534 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.053535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.053540 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.053599 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.054817 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.054819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.054819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.054823 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.054840 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.054844 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.054917 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.054989 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.055002 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:05:04.055222 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.052746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43963 (local address 127.0.98.3:59653)
0926 01:05:04.052926 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.052979 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:04.053449 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:04.053651 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:04.053655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:04.053664 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:04.054621 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:04.054622 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:04.054653 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:04.054692 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:04.054922 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:04.054925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:04.054928 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:04.054982 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 01:05:04.055035 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260926 01:05:04.055754 5450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43963
I20260926 01:05:04.055843 5450 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:04.055959 5450 heartbeater.cc:511] Master 127.0.98.62:43963 requested a full tablet report, sending...
I20260926 01:05:04.056244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 5322
I20260926 01:05:04.056346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:05:04.056394 5000 ts_manager.cc:194] Registered new tserver with Master: 02bc4f2c0e514894957c977c6f04f5c8 (127.0.98.3:45621)
I20260926 01:05:04.056520 392 external_mini_cluster.cc:1428] Setting key 499d080c07813672582407b6d8622abb
I20260926 01:05:04.057435 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:05:04.059876 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.058892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.0.98.62:43963)
0926 01:05:04.058935 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.058937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.058991 (+ 54us) server_negotiation.cc:407] Connection header received
0926 01:05:04.059040 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.059041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.059045 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.059095 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.059646 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.059647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.059647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.059650 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.059665 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.059669 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.059712 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.059811 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.059823 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:05:04.061466 5000 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:05:04.063091 5000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:05:04.066983 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.065336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.0.98.3:45621)
0926 01:05:04.065768 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.065772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.065781 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:04.065805 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.065807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.065815 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.065885 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.066605 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.066613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.066613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.066617 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.066651 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.066682 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.066735 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.066881 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.066898 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":95,"threads_started":1}
I20260926 01:05:04.067000 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.065454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36395 (local address 127.0.0.1:55788)
0926 01:05:04.065610 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.065619 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:04.065648 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:04.066327 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:04.066328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:04.066330 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:04.066669 (+ 339us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:04.066669 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:04.066673 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:04.066681 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:04.066911 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:04.066912 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:04.066912 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 01:05:04.066936 (+ 24us) client_negotiation.cc:240] Negotiation successful
0926 01:05:04.066946 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
I20260926 01:05:04.067157 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.065239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45621 (local address 127.0.0.1:56092)
0926 01:05:04.065447 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.065489 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:04.065542 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:04.065889 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:04.065890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:04.065893 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:04.066516 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:04.066517 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:04.066523 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:04.066534 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:04.066761 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:04.066767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:04.066770 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:04.066829 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:05:04.066842 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":66,"threads_started":1}
I20260926 01:05:04.067229 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.065681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.0.98.2:36395)
0926 01:05:04.066205 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.066208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.066216 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:04.066236 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.066237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.066244 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.066307 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.066787 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.066793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.066793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.066797 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.066826 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.066845 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.067103 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.067127 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.067142 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":105,"threads_started":1}
I20260926 01:05:04.067230 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.065373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.0.98.1:42779)
0926 01:05:04.065682 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.065687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.065697 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:04.065721 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.065723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.065730 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.065790 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.066623 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.066627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.066627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.066631 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.066666 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.066682 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.066739 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.066993 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.067011 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":70,"threads_started":1}
I20260926 01:05:04.067339 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.065246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42779 (local address 127.0.0.1:44636)
0926 01:05:04.065559 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.065570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:04.065615 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:04.065802 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:04.065808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:04.065815 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:04.066333 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:04.066334 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:04.066353 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:04.066401 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:04.066762 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:04.066767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:04.066770 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:04.066826 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:05:04.066840 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":88,"threads_started":1}
I20260926 01:05:04.067838 5253 tablet_service.cc:1467] Processing CreateTablet for tablet b0e1e22ec5ea4dafac38f570cb102d4e (DEFAULT_TABLE table=test-table [id=fc59f63b352e46b68cb2b576909e89a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:04.068058 5122 tablet_service.cc:1467] Processing CreateTablet for tablet b0e1e22ec5ea4dafac38f570cb102d4e (DEFAULT_TABLE table=test-table [id=fc59f63b352e46b68cb2b576909e89a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:04.068220 5253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e1e22ec5ea4dafac38f570cb102d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:04.068353 5122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e1e22ec5ea4dafac38f570cb102d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:04.070823 5384 tablet_service.cc:1467] Processing CreateTablet for tablet b0e1e22ec5ea4dafac38f570cb102d4e (DEFAULT_TABLE table=test-table [id=fc59f63b352e46b68cb2b576909e89a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:04.071185 5384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e1e22ec5ea4dafac38f570cb102d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:04.071600 5465 tablet_bootstrap.cc:492] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0: Bootstrap starting.
I20260926 01:05:04.071779 5464 tablet_bootstrap.cc:492] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49: Bootstrap starting.
I20260926 01:05:04.072268 5464 tablet_bootstrap.cc:654] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:04.072273 5465 tablet_bootstrap.cc:654] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:04.072479 5464 log.cc:824] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:04.072566 5465 log.cc:824] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:04.073315 5465 tablet_bootstrap.cc:492] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0: No bootstrap required, opened a new log
I20260926 01:05:04.073315 5464 tablet_bootstrap.cc:492] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49: No bootstrap required, opened a new log
I20260926 01:05:04.073377 5465 ts_tablet_manager.cc:1397] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:04.073378 5464 ts_tablet_manager.cc:1397] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:04.074532 5468 tablet_bootstrap.cc:492] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: Bootstrap starting.
I20260926 01:05:04.075336 5468 tablet_bootstrap.cc:654] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:04.075659 5468 log.cc:824] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:04.075703 5464 raft_consensus.cc:348] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 } } peers { permanent_uuid: "032b5410fd74415c818ca00da29526d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42779 } } peers { permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 } }
I20260926 01:05:04.075752 5465 raft_consensus.cc:348] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 } } peers { permanent_uuid: "032b5410fd74415c818ca00da29526d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42779 } } peers { permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 } }
I20260926 01:05:04.075871 5464 raft_consensus.cc:374] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:04.075901 5465 raft_consensus.cc:374] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:04.075913 5464 raft_consensus.cc:729] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11787f4e7de450dba0ea190bd20af49, State: Initialized, Role: FOLLOWER
I20260926 01:05:04.075932 5465 raft_consensus.cc:729] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032b5410fd74415c818ca00da29526d0, State: Initialized, Role: FOLLOWER
I20260926 01:05:04.076026 5464 consensus_queue.cc:260] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 } } peers { permanent_uuid: "032b5410fd74415c818ca00da29526d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42779 } } peers { permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 } }
I20260926 01:05:04.076053 5465 consensus_queue.cc:260] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 } } peers { permanent_uuid: "032b5410fd74415c818ca00da29526d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42779 } } peers { permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 } }
I20260926 01:05:04.076299 5464 ts_tablet_manager.cc:1428] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:05:04.076340 5319 heartbeater.cc:503] Master 127.0.98.62:43963 was elected leader, sending a full tablet report...
I20260926 01:05:04.076298 5465 ts_tablet_manager.cc:1428] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:05:04.076452 5188 heartbeater.cc:503] Master 127.0.98.62:43963 was elected leader, sending a full tablet report...
I20260926 01:05:04.076889 5468 tablet_bootstrap.cc:492] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: No bootstrap required, opened a new log
I20260926 01:05:04.076959 5468 ts_tablet_manager.cc:1397] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:04.079028 5468 raft_consensus.cc:348] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 } } peers { permanent_uuid: "032b5410fd74415c818ca00da29526d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42779 } } peers { permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 } }
I20260926 01:05:04.079180 5468 raft_consensus.cc:374] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:04.079218 5468 raft_consensus.cc:729] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bc4f2c0e514894957c977c6f04f5c8, State: Initialized, Role: FOLLOWER
I20260926 01:05:04.079358 5468 consensus_queue.cc:260] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 } } peers { permanent_uuid: "032b5410fd74415c818ca00da29526d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42779 } } peers { permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 } }
I20260926 01:05:04.079612 5468 ts_tablet_manager.cc:1428] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:05:04.079686 5450 heartbeater.cc:503] Master 127.0.98.62:43963 was elected leader, sending a full tablet report...
I20260926 01:05:04.116616 5471 raft_consensus.cc:482] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:04.116751 5471 raft_consensus.cc:504] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 } } peers { permanent_uuid: "032b5410fd74415c818ca00da29526d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42779 } } peers { permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 } }
I20260926 01:05:04.117091 5471 leader_election.cc:288] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02bc4f2c0e514894957c977c6f04f5c8 (127.0.98.3:45621), b11787f4e7de450dba0ea190bd20af49 (127.0.98.2:36395)
I20260926 01:05:04.118292 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.117272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36395 (local address 127.0.98.1:47929)
0926 01:05:04.117323 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.117336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:04.117377 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:04.117513 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:04.117514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:04.117517 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:04.117942 (+ 425us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:04.117942 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:04.117950 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:04.117964 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:04.118154 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:04.118155 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:04.118156 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:05:04.118211 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:05:04.118226 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:05:04.118307 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.117346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47929 (local address 127.0.98.2:36395)
0926 01:05:04.117388 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.117390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.117400 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:04.117428 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.117429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.117435 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.117496 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.118047 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.118049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.118049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.118053 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.118072 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.118078 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.118138 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.118222 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.118236 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:05:04.118592 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.117414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58747 (local address 127.0.98.3:45621)
0926 01:05:04.117471 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.117473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.117545 (+ 72us) server_negotiation.cc:407] Connection header received
0926 01:05:04.117570 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.117571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.117575 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.117620 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.118228 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.118229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.118229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.118233 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.118248 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.118253 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.118298 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.118511 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.118527 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:05:04.118717 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e1e22ec5ea4dafac38f570cb102d4e" candidate_uuid: "032b5410fd74415c818ca00da29526d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b11787f4e7de450dba0ea190bd20af49" is_pre_election: true
I20260926 01:05:04.118785 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.117290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45621 (local address 127.0.98.1:58747)
0926 01:05:04.117470 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.117478 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:04.117517 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:04.117629 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:04.117630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:04.117632 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:04.118096 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:04.118096 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:04.118102 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:04.118114 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:04.118303 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:04.118303 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:04.118304 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:05:04.118696 (+ 392us) client_negotiation.cc:240] Negotiation successful
0926 01:05:04.118710 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":108,"threads_started":1}
I20260926 01:05:04.118878 5273 raft_consensus.cc:2454] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032b5410fd74415c818ca00da29526d0 in term 0.
I20260926 01:05:04.119096 5076 leader_election.cc:302] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032b5410fd74415c818ca00da29526d0, b11787f4e7de450dba0ea190bd20af49; no voters:
I20260926 01:05:04.119196 5404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e1e22ec5ea4dafac38f570cb102d4e" candidate_uuid: "032b5410fd74415c818ca00da29526d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bc4f2c0e514894957c977c6f04f5c8" is_pre_election: true
I20260926 01:05:04.119261 5471 raft_consensus.cc:2781] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:05:04.119338 5471 raft_consensus.cc:482] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:04.119338 5404 raft_consensus.cc:2454] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032b5410fd74415c818ca00da29526d0 in term 0.
I20260926 01:05:04.119371 5471 raft_consensus.cc:3037] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:04.120119 5471 raft_consensus.cc:504] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 } } peers { permanent_uuid: "032b5410fd74415c818ca00da29526d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42779 } } peers { permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 } }
I20260926 01:05:04.120254 5471 leader_election.cc:288] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [CANDIDATE]: Term 1 election: Requested vote from peers 02bc4f2c0e514894957c977c6f04f5c8 (127.0.98.3:45621), b11787f4e7de450dba0ea190bd20af49 (127.0.98.2:36395)
I20260926 01:05:04.120445 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e1e22ec5ea4dafac38f570cb102d4e" candidate_uuid: "032b5410fd74415c818ca00da29526d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b11787f4e7de450dba0ea190bd20af49"
I20260926 01:05:04.120477 5404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e1e22ec5ea4dafac38f570cb102d4e" candidate_uuid: "032b5410fd74415c818ca00da29526d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bc4f2c0e514894957c977c6f04f5c8"
I20260926 01:05:04.120540 5273 raft_consensus.cc:3037] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:04.120543 5404 raft_consensus.cc:3037] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:04.121429 5273 raft_consensus.cc:2454] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032b5410fd74415c818ca00da29526d0 in term 1.
I20260926 01:05:04.121428 5404 raft_consensus.cc:2454] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032b5410fd74415c818ca00da29526d0 in term 1.
I20260926 01:05:04.121639 5076 leader_election.cc:302] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032b5410fd74415c818ca00da29526d0, b11787f4e7de450dba0ea190bd20af49; no voters:
I20260926 01:05:04.121742 5471 raft_consensus.cc:2781] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:04.121922 5471 raft_consensus.cc:686] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [term 1 LEADER]: Becoming Leader. State: Replica: 032b5410fd74415c818ca00da29526d0, State: Running, Role: LEADER
I20260926 01:05:04.122014 5471 consensus_queue.cc:237] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 } } peers { permanent_uuid: "032b5410fd74415c818ca00da29526d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42779 } } peers { permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 } }
I20260926 01:05:04.122903 5000 catalog_manager.cc:5495] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 032b5410fd74415c818ca00da29526d0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "032b5410fd74415c818ca00da29526d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032b5410fd74415c818ca00da29526d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 } health_report { overall_health: UNKNOWN } } }
I20260926 01:05:04.130515 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.129595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.0.98.1:42779)
0926 01:05:04.129641 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.129643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.129653 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:04.129736 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.129737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.129751 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.129798 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.130295 (+ 497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.130296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.130297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.130299 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.130311 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.130314 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.130349 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.130408 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.130421 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:05:04.132397 5273 raft_consensus.cc:1261] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49 [term 1 FOLLOWER]: Refusing update from remote peer 032b5410fd74415c818ca00da29526d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:04.132397 5404 raft_consensus.cc:1261] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8 [term 1 FOLLOWER]: Refusing update from remote peer 032b5410fd74415c818ca00da29526d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:04.132738 5471 consensus_queue.cc:1035] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02bc4f2c0e514894957c977c6f04f5c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:04.132886 5474 consensus_queue.cc:1035] T b0e1e22ec5ea4dafac38f570cb102d4e P 032b5410fd74415c818ca00da29526d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b11787f4e7de450dba0ea190bd20af49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:04.135401 5477 mvcc.cc:205] Tried to move back new op lower bound from 7333415748123529216 to 7333415748084322304. Current Snapshot: MvccSnapshot[applied={T|T < 7333415748123529216}]
I20260926 01:05:04.136041 5479 mvcc.cc:205] Tried to move back new op lower bound from 7333415748123529216 to 7333415748084322304. Current Snapshot: MvccSnapshot[applied={T|T < 7333415748123529216 or (T in {7333415748123529216})}]
I20260926 01:05:04.136704 5478 mvcc.cc:205] Tried to move back new op lower bound from 7333415748123529216 to 7333415748084322304. Current Snapshot: MvccSnapshot[applied={T|T < 7333415748123529216 or (T in {7333415748123529216})}]
I20260926 01:05:04.137945 5000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:04.138044 5000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:04.138775 5000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d78e0ddf9e0462dae6c7049f777ed32: Sending DeleteTablet for 3 replicas of tablet b0e1e22ec5ea4dafac38f570cb102d4e
I20260926 01:05:04.139115 5122 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e1e22ec5ea4dafac38f570cb102d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:04 UTC) from {username='slave'} at 127.0.0.1:44636
I20260926 01:05:04.139153 5384 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e1e22ec5ea4dafac38f570cb102d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:04 UTC) from {username='slave'} at 127.0.0.1:56092
I20260926 01:05:04.139185 5253 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e1e22ec5ea4dafac38f570cb102d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:04 UTC) from {username='slave'} at 127.0.0.1:55788
I20260926 01:05:04.139552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 5060
I20260926 01:05:04.142546 5487 tablet_replica.cc:330] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: stopping tablet replica
I20260926 01:05:04.142736 5487 raft_consensus.cc:2229] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:04.142869 5487 raft_consensus.cc:2258] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:04.143049 5488 tablet_replica.cc:330] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49: stopping tablet replica
I20260926 01:05:04.143162 5488 raft_consensus.cc:2229] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:04.143383 5488 raft_consensus.cc:2258] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:04.143486 5487 ts_tablet_manager.cc:1905] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:04.143858 5488 ts_tablet_manager.cc:1905] T b0e1e22ec5ea4dafac38f570cb102d4e P b11787f4e7de450dba0ea190bd20af49: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:05:04.144918 4985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42779 (error 108)
I20260926 01:05:04.145071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 5191
I20260926 01:05:04.145740 5487 ts_tablet_manager.cc:1918] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:04.145821 5487 log.cc:1196] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/b0e1e22ec5ea4dafac38f570cb102d4e
I20260926 01:05:04.146034 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.145916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42779 (local address 127.0.0.1:44654)
0926 01:05:04.145964 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.146000 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:05:04.146067 5487 ts_tablet_manager.cc:1939] T b0e1e22ec5ea4dafac38f570cb102d4e P 02bc4f2c0e514894957c977c6f04f5c8: Deleting consensus metadata
W20260926 01:05:04.146150 4985 catalog_manager.cc:4584] TS 032b5410fd74415c818ca00da29526d0 (127.0.98.1:42779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0e1e22ec5ea4dafac38f570cb102d4e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42779: connect: Connection refused (error 111)
I20260926 01:05:04.146502 4986 catalog_manager.cc:4841] TS 02bc4f2c0e514894957c977c6f04f5c8 (127.0.98.3:45621): tablet b0e1e22ec5ea4dafac38f570cb102d4e (table test-table [id=fc59f63b352e46b68cb2b576909e89a6]) successfully deleted
I20260926 01:05:04.149667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 5322
I20260926 01:05:04.149780 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.149646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36395 (local address 127.0.0.1:55798)
0926 01:05:04.149703 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.149736 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:05:04.154635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 4969
2026-09-26T01:05:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (654 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 01:05:04.173694 392 mini_oidc.cc:152] Starting JWKS server
I20260926 01:05:04.175025 392 webserver.cc:460] Webserver started at https://127.0.0.1:41097/ using document root <none> and password file <none>
I20260926 01:05:04.175127 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 01:05:04.175287 392 webserver.cc:460] Webserver started at http://127.0.0.1:40953/ using document root <none> and password file <none>
2026-09-26T01:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:04Z Disabled control of system clock
I20260926 01:05:04.182931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45985
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45985
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/testchainca.cert with env {}
W20260926 01:05:04.266902 5496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:04.267139 5496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:04.267168 5496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:04.268446 5496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:04.268505 5496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:04.268519 5496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:04.268533 5496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:04.268545 5496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 01:05:04.268559 5496 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 01:05:04.268571 5496 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41097/jwks.json
I20260926 01:05:04.270236 5496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:04.270535 5496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:04.270745 5496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:04.275880 5502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:04.276052 5501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:04.275885 5504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:04.276141 5496 server_base.cc:1034] running on GCE node
I20260926 01:05:04.276427 5496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:04.276723 5496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:04.277897 5496 hybrid_clock.cc:648] HybridClock initialized: now 1790384704277880 us; error 40 us; skew 500 ppm
I20260926 01:05:04.279399 5496 webserver.cc:460] Webserver started at http://127.0.98.62:33789/ using document root <none> and password file <none>
I20260926 01:05:04.279733 5496 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:04.279819 5496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:04.279963 5496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:04.281049 5496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "e4c9d2bec7874c7c8f14b40fea5afd33"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "e0f7b76597185fdfaaa804ef5b039667"
server_key_iv: "f010f285a5c3804f1e7ffa328c9df3bf"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.281389 5496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4c9d2bec7874c7c8f14b40fea5afd33"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "e0f7b76597185fdfaaa804ef5b039667"
server_key_iv: "f010f285a5c3804f1e7ffa328c9df3bf"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.282823 5496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:04.283784 5510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:04.284070 5496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:04.284171 5496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "e4c9d2bec7874c7c8f14b40fea5afd33"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "e0f7b76597185fdfaaa804ef5b039667"
server_key_iv: "f010f285a5c3804f1e7ffa328c9df3bf"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.284250 5496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:04.299635 5496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:04.299966 5496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:04.300096 5496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:04.304026 5496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45985
I20260926 01:05:04.304081 5562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45985 every 8 connection(s)
I20260926 01:05:04.304404 5496 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:04.305469 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:04.307575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 5496
I20260926 01:05:04.307670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:04.307857 392 external_mini_cluster.cc:1428] Setting key cadd9d4fbd3275f580822ec57129bc4d
I20260926 01:05:04.308091 5563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33: Bootstrap starting.
I20260926 01:05:04.309083 5563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:04.309432 5563 log.cc:824] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:04.310226 5563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33: No bootstrap required, opened a new log
I20260926 01:05:04.312572 5563 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c9d2bec7874c7c8f14b40fea5afd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45985 } }
I20260926 01:05:04.312781 5563 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:04.312817 5563 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c9d2bec7874c7c8f14b40fea5afd33, State: Initialized, Role: FOLLOWER
I20260926 01:05:04.312949 5563 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c9d2bec7874c7c8f14b40fea5afd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45985 } }
I20260926 01:05:04.313032 5563 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:04.313076 5563 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:04.313136 5563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:04.313979 5563 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c9d2bec7874c7c8f14b40fea5afd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45985 } }
I20260926 01:05:04.314083 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.308321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.0.98.62:45985)
0926 01:05:04.308518 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.308527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.308561 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:05:04.309142 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.309153 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.309269 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.309434 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.309631 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.310925 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.311779 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.312471 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.312604 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:04.313304 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.313309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.313309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.313527 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.313573 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.313588 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.313667 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.313781 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.313871 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260926 01:05:04.314098 5563 leader_election.cc:302] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4c9d2bec7874c7c8f14b40fea5afd33; no voters:
I20260926 01:05:04.314548 5563 leader_election.cc:288] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:04.314590 5568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:04.314842 5563 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:04.314900 5568 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [term 1 LEADER]: Becoming Leader. State: Replica: e4c9d2bec7874c7c8f14b40fea5afd33, State: Running, Role: LEADER
I20260926 01:05:04.315023 5568 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c9d2bec7874c7c8f14b40fea5afd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45985 } }
I20260926 01:05:04.315171 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:05:04.316159 5570 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4c9d2bec7874c7c8f14b40fea5afd33. Latest consensus state: current_term: 1 leader_uuid: "e4c9d2bec7874c7c8f14b40fea5afd33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c9d2bec7874c7c8f14b40fea5afd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45985 } } }
I20260926 01:05:04.316179 5569 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4c9d2bec7874c7c8f14b40fea5afd33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c9d2bec7874c7c8f14b40fea5afd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45985 } } }
I20260926 01:05:04.316452 5570 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:04.316457 5569 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33 [sys.catalog]: This master's current role is: LEADER
W20260926 01:05:04.316859 5582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:05:04.316912 5582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:05:04.317126 5583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:04.317286 5583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:04.318807 5583 catalog_manager.cc:1347] Generated new cluster ID: 3624e462c9b64ce0abd993fb3178da1e
I20260926 01:05:04.318868 5583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:04.323498 5583 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:04.324163 5583 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:04.332041 5583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4c9d2bec7874c7c8f14b40fea5afd33: Generated new TSK 0
I20260926 01:05:04.332270 5583 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 01:05:04.362046 5593 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 01:05:04.362228 5593 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 01:05:04.362424 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.339620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.0.98.62:45985)
0926 01:05:04.339666 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.339668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.339793 (+ 125us) server_negotiation.cc:407] Connection header received
0926 01:05:04.339908 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.339909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.339919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.339977 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0926 01:05:04.340204 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.340773 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.341905 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.342107 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.342168 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:04.342469 (+ 301us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 01:05:04.362017 (+ 19548us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 01:05:04.362261 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38444: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 01:05:04.362516 5566 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38444: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 01:05:04.362684 5592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:04.339513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45985 (local address 127.0.0.1:38444)
0926 01:05:04.339698 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.339712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:04.339801 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:04.339996 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:04.339997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:04.340000 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 01:05:04.340098 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:04.340101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.340843 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.340845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:04.341796 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:04.341800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.342191 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.342195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:04.342261 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:04.342293 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 01:05:04.362186 (+ 19893us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 01:05:04.362316 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45985: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:45985: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 01:05:04.363175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 5496
2026-09-26T01:05:04Z chronyd exiting
I20260926 01:05:04.399036 392 test_util.cc:175] -----------------------------------------------
I20260926 01:05:04.399138 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790384680280236-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 01:05:04.399931 392 mini_oidc.cc:152] Starting JWKS server
I20260926 01:05:04.400588 392 webserver.cc:460] Webserver started at https://127.0.0.1:39385/ using document root <none> and password file <none>
I20260926 01:05:04.400628 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 01:05:04.400740 392 webserver.cc:460] Webserver started at http://127.0.0.1:46783/ using document root <none> and password file <none>
2026-09-26T01:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:04Z Disabled control of system clock
I20260926 01:05:04.408689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38791
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38791
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39385/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/testchainca.cert with env {}
W20260926 01:05:04.491818 5599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:04.492070 5599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:04.492097 5599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:04.493543 5599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:04.493599 5599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:04.493610 5599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:04.493623 5599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:04.493634 5599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 01:05:04.493645 5599 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 01:05:04.493655 5599 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39385/jwks.json
I20260926 01:05:04.495515 5599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39385/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:04.495816 5599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:04.496037 5599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:04.501540 5605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:04.501636 5604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:04.501717 5599 server_base.cc:1034] running on GCE node
W20260926 01:05:04.501535 5607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:04.502067 5599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:04.502401 5599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:04.503600 5599 hybrid_clock.cc:648] HybridClock initialized: now 1790384704503577 us; error 35 us; skew 500 ppm
I20260926 01:05:04.505023 5599 webserver.cc:460] Webserver started at http://127.0.98.62:41731/ using document root <none> and password file <none>
I20260926 01:05:04.505264 5599 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:04.505316 5599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:04.505435 5599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:04.506564 5599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "2db95900090243c290823db803f8c4a0"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "4db7e679058ce4570f57252047c788cf"
server_key_iv: "cb29e82fbd8d108d13f65da4a88cf6b1"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.506912 5599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "2db95900090243c290823db803f8c4a0"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "4db7e679058ce4570f57252047c788cf"
server_key_iv: "cb29e82fbd8d108d13f65da4a88cf6b1"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.508250 5599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:04.509253 5613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:04.509580 5599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:04.509691 5599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "2db95900090243c290823db803f8c4a0"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "4db7e679058ce4570f57252047c788cf"
server_key_iv: "cb29e82fbd8d108d13f65da4a88cf6b1"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.509771 5599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:04.526516 5599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:04.526877 5599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:04.527015 5599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:04.530880 5599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38791
I20260926 01:05:04.530922 5665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38791 every 8 connection(s)
I20260926 01:05:04.531260 5599 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:04.532188 5666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:04.533347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 5599
I20260926 01:05:04.533447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:04.533607 392 external_mini_cluster.cc:1428] Setting key 679dcc532fa6ce7d257d0f0a6deda2e5
I20260926 01:05:04.534951 5666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0: Bootstrap starting.
I20260926 01:05:04.535593 5666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:04.535851 5666 log.cc:824] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:04.536646 5666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0: No bootstrap required, opened a new log
I20260926 01:05:04.538959 5666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db95900090243c290823db803f8c4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } }
I20260926 01:05:04.539126 5666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:04.539163 5666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db95900090243c290823db803f8c4a0, State: Initialized, Role: FOLLOWER
I20260926 01:05:04.539315 5666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db95900090243c290823db803f8c4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } }
I20260926 01:05:04.539409 5666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:04.539433 5666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:04.539456 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.534152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.0.98.62:38791)
0926 01:05:04.534358 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.534366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.534421 (+ 55us) server_negotiation.cc:407] Connection header received
0926 01:05:04.534829 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.534836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.534900 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.535048 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.535280 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.536337 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.537409 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.537872 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.537973 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:04.538765 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.538769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.538769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.538937 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.538969 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.538980 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.539036 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.539131 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.539233 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260926 01:05:04.539479 5666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:04.540200 5666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db95900090243c290823db803f8c4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } }
I20260926 01:05:04.540309 5666 leader_election.cc:302] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2db95900090243c290823db803f8c4a0; no voters:
I20260926 01:05:04.540490 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:05:04.540508 5666 leader_election.cc:288] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:04.540530 5671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:04.540624 5671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [term 1 LEADER]: Becoming Leader. State: Replica: 2db95900090243c290823db803f8c4a0, State: Running, Role: LEADER
I20260926 01:05:04.540704 5671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db95900090243c290823db803f8c4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } }
I20260926 01:05:04.540752 5666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:04.541018 5671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2db95900090243c290823db803f8c4a0. Latest consensus state: current_term: 1 leader_uuid: "2db95900090243c290823db803f8c4a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db95900090243c290823db803f8c4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } } }
I20260926 01:05:04.541031 5672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2db95900090243c290823db803f8c4a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db95900090243c290823db803f8c4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } } }
I20260926 01:05:04.541224 5672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:04.541095 5671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:04.542073 5679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:04.542557 5679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:04.544100 5679 catalog_manager.cc:1347] Generated new cluster ID: ddc95710d9b142f392d94fafc796ef64
I20260926 01:05:04.544154 5679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:04.595451 5679 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:04.596179 5679 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:04.603924 5679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2db95900090243c290823db803f8c4a0: Generated new TSK 0
I20260926 01:05:04.604141 5679 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 01:05:04.641021 5696 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 01:05:04.641119 5696 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260926 01:05:04.641448 5695 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:04.620087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38791 (local address 127.0.0.1:44388)
0926 01:05:04.620246 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.620258 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:04.620362 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:04.620518 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:04.620518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:04.620520 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 01:05:04.620581 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:04.620584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.621380 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.621381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:04.622007 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:04.622009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.622352 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.622356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:04.622444 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:04.622451 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 01:05:04.641252 (+ 18801us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 01:05:04.641344 (+ 92us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260926 01:05:04.641570 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.620148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.0.98.62:38791)
0926 01:05:04.620200 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.620202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.620354 (+ 152us) server_negotiation.cc:407] Connection header received
0926 01:05:04.620442 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.620444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.620454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.620512 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0926 01:05:04.620638 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.621248 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.622089 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.622289 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.622343 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:04.622544 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 01:05:04.641136 (+ 18592us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 01:05:04.641492 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 01:05:04.641616 5669 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 01:05:04.641748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 5599
2026-09-26T01:05:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (280 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 01:05:04.680984 392 mini_oidc.cc:152] Starting JWKS server
I20260926 01:05:04.681659 392 webserver.cc:460] Webserver started at https://127.0.0.1:43103/ using document root <none> and password file <none>
I20260926 01:05:04.681702 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 01:05:04.681840 392 webserver.cc:460] Webserver started at http://127.0.0.1:40169/ using document root <none> and password file <none>
2026-09-26T01:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:04Z Disabled control of system clock
I20260926 01:05:04.690133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41747
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41747
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43103/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 01:05:04.773201 5702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:04.773450 5702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:04.773483 5702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:04.774811 5702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:04.774863 5702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:04.774880 5702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:04.774895 5702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:04.774914 5702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 01:05:04.774937 5702 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 01:05:04.774953 5702 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43103/jwks.json
I20260926 01:05:04.776507 5702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43103/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:04.776783 5702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:04.777078 5702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:04.782095 5710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:04.782107 5708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:04.782292 5702 server_base.cc:1034] running on GCE node
W20260926 01:05:04.782126 5707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:04.782632 5702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:04.782998 5702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:04.784173 5702 hybrid_clock.cc:648] HybridClock initialized: now 1790384704784153 us; error 41 us; skew 500 ppm
I20260926 01:05:04.785590 5702 webserver.cc:460] Webserver started at http://127.0.98.62:38747/ using document root <none> and password file <none>
I20260926 01:05:04.785837 5702 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:04.785888 5702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:04.785995 5702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:04.787171 5702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "d22469f8ac4749e2bda09e2a2a09b5de"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "d4a4fa14e21d1ae4a0068cdbaa6c3448"
server_key_iv: "875915346104ada74ebd68b4217d63b4"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.787540 5702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "d22469f8ac4749e2bda09e2a2a09b5de"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "d4a4fa14e21d1ae4a0068cdbaa6c3448"
server_key_iv: "875915346104ada74ebd68b4217d63b4"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.788851 5702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:04.789717 5716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:04.789961 5702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:04.790050 5702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "d22469f8ac4749e2bda09e2a2a09b5de"
format_stamp: "Formatted at 2026-09-26 01:05:04 on dist-test-slave-l73v"
server_key: "d4a4fa14e21d1ae4a0068cdbaa6c3448"
server_key_iv: "875915346104ada74ebd68b4217d63b4"
server_key_version: "encryptionkey@0"
I20260926 01:05:04.790127 5702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:04.815748 5702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:04.816068 5702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:04.816198 5702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:04.820055 5702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41747
I20260926 01:05:04.820138 5768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41747 every 8 connection(s)
I20260926 01:05:04.820425 5702 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:04.821126 5769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:04.823931 5769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de: Bootstrap starting.
I20260926 01:05:04.824574 5769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:04.824862 5769 log.cc:824] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:04.825148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 5702
I20260926 01:05:04.825232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:04.825438 392 external_mini_cluster.cc:1428] Setting key fe8ed03ec83730ce8a2ca6f180461e62
I20260926 01:05:04.825681 5769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de: No bootstrap required, opened a new log
I20260926 01:05:04.827831 5769 raft_consensus.cc:348] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22469f8ac4749e2bda09e2a2a09b5de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41747 } }
I20260926 01:05:04.828006 5769 raft_consensus.cc:374] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:04.828042 5769 raft_consensus.cc:729] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22469f8ac4749e2bda09e2a2a09b5de, State: Initialized, Role: FOLLOWER
I20260926 01:05:04.828191 5769 consensus_queue.cc:260] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22469f8ac4749e2bda09e2a2a09b5de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41747 } }
I20260926 01:05:04.828329 5769 raft_consensus.cc:388] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:04.828378 5769 raft_consensus.cc:482] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:04.828428 5769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:04.829294 5769 raft_consensus.cc:504] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22469f8ac4749e2bda09e2a2a09b5de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41747 } }
I20260926 01:05:04.829391 5769 leader_election.cc:302] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d22469f8ac4749e2bda09e2a2a09b5de; no voters:
I20260926 01:05:04.829577 5769 leader_election.cc:288] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:04.829622 5774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:04.829730 5774 raft_consensus.cc:686] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [term 1 LEADER]: Becoming Leader. State: Replica: d22469f8ac4749e2bda09e2a2a09b5de, State: Running, Role: LEADER
I20260926 01:05:04.829823 5774 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22469f8ac4749e2bda09e2a2a09b5de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41747 } }
I20260926 01:05:04.829864 5769 sys_catalog.cc:564] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:04.830148 5776 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22469f8ac4749e2bda09e2a2a09b5de. Latest consensus state: current_term: 1 leader_uuid: "d22469f8ac4749e2bda09e2a2a09b5de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22469f8ac4749e2bda09e2a2a09b5de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41747 } } }
I20260926 01:05:04.830224 5776 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:04.830375 5775 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d22469f8ac4749e2bda09e2a2a09b5de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22469f8ac4749e2bda09e2a2a09b5de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41747 } } }
I20260926 01:05:04.830464 5775 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:04.831017 5779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:04.831955 5779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:04.832753 5773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.825996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.0.98.62:41747)
0926 01:05:04.826216 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.826265 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.826301 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:05:04.827819 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.827829 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.827940 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.828116 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:04.828367 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.829422 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.830522 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.831147 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.831273 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:04.831982 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:04.831987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:04.831987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:04.832203 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:04.832247 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:04.832263 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:04.832343 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:04.832419 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 01:05:04.832506 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260926 01:05:04.833748 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:05:04.834410 5779 catalog_manager.cc:1347] Generated new cluster ID: 12550a3e18dc4c3faef3b452d521a9c4
I20260926 01:05:04.834468 5779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:04.842698 5779 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:04.843307 5779 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:04.853225 5779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d22469f8ac4749e2bda09e2a2a09b5de: Generated new TSK 0
I20260926 01:05:04.853494 5779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:05:04.871574 5773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:04.866963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.0.98.62:41747)
0926 01:05:04.867032 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:04.867035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:04.867099 (+ 64us) server_negotiation.cc:407] Connection header received
0926 01:05:04.867190 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:04.867190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:04.867203 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:04.867288 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0926 01:05:04.867419 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.868163 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.869378 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.869584 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.869647 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:04.869921 (+ 274us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 01:05:04.871346 (+ 1425us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 01:05:04.871493 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 01:05:04.871667 5798 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:04.866856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41747 (local address 127.0.0.1:38234)
0926 01:05:04.867030 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:04.867041 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:04.867119 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:04.867293 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:04.867294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:04.867297 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 01:05:04.867360 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:04.867363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.868273 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.868274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:04.869250 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:04.869254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:04.869660 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:04.869668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:04.869730 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:04.869736 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 01:05:04.871464 (+ 1728us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 01:05:04.871571 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
W20260926 01:05:04.871827 5773 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43103/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 01:05:04.872164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 5702
E20260926 01:05:04.882758 5799 webserver.cc:533] Webserver: sslize failed:
E20260926 01:05:04.882846 5799 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T01:05:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T01:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:04Z Disabled control of system clock
I20260926 01:05:04.916545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36457
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36457
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/testchainca.cert with env {}
W20260926 01:05:04.997370 5803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:04.997599 5803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:04.997629 5803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:04.998885 5803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:04.998931 5803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:04.998948 5803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:04.998965 5803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:04.998979 5803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 01:05:04.998996 5803 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 01:05:04.999014 5803 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 01:05:05.000859 5803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:05.001160 5803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:05.001379 5803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:05:05.006683 5803 server_base.cc:1034] running on GCE node
W20260926 01:05:05.006687 5811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:05.006672 5809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:05.006670 5808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:05.007102 5803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:05.007408 5803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:05.008554 5803 hybrid_clock.cc:648] HybridClock initialized: now 1790384705008536 us; error 36 us; skew 500 ppm
I20260926 01:05:05.009920 5803 webserver.cc:460] Webserver started at http://127.0.98.62:41833/ using document root <none> and password file <none>
I20260926 01:05:05.010154 5803 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:05.010200 5803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:05.010322 5803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:05.011490 5803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "916887cd133a48868c19fd3356296ff3"
format_stamp: "Formatted at 2026-09-26 01:05:05 on dist-test-slave-l73v"
server_key: "da5b719da8548dda7992ab012e0dc820"
server_key_iv: "6f4cb74b955613af61e49ea763795f31"
server_key_version: "encryptionkey@0"
I20260926 01:05:05.011859 5803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "916887cd133a48868c19fd3356296ff3"
format_stamp: "Formatted at 2026-09-26 01:05:05 on dist-test-slave-l73v"
server_key: "da5b719da8548dda7992ab012e0dc820"
server_key_iv: "6f4cb74b955613af61e49ea763795f31"
server_key_version: "encryptionkey@0"
I20260926 01:05:05.013165 5803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:05.013962 5817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:05.014139 5803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:05.014227 5803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "916887cd133a48868c19fd3356296ff3"
format_stamp: "Formatted at 2026-09-26 01:05:05 on dist-test-slave-l73v"
server_key: "da5b719da8548dda7992ab012e0dc820"
server_key_iv: "6f4cb74b955613af61e49ea763795f31"
server_key_version: "encryptionkey@0"
I20260926 01:05:05.014283 5803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:05.050310 5803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:05.050640 5803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:05.050747 5803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:05.054263 5803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36457
I20260926 01:05:05.054298 5869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36457 every 8 connection(s)
I20260926 01:05:05.054646 5803 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:05.055615 5870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:05.057920 5870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3: Bootstrap starting.
I20260926 01:05:05.058535 5870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:05.058789 5870 log.cc:824] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:05.059536 5870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3: No bootstrap required, opened a new log
I20260926 01:05:05.061129 5870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916887cd133a48868c19fd3356296ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } }
I20260926 01:05:05.061285 5870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:05.061318 5870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916887cd133a48868c19fd3356296ff3, State: Initialized, Role: FOLLOWER
I20260926 01:05:05.061487 5870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916887cd133a48868c19fd3356296ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } }
I20260926 01:05:05.061528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 5803
I20260926 01:05:05.061566 5870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:05.061596 5870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:05.061619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:05.061635 5870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:05.061795 392 external_mini_cluster.cc:1428] Setting key f0715bb7827ea7f053b8812b0427e20a
I20260926 01:05:05.062297 5870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916887cd133a48868c19fd3356296ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } }
I20260926 01:05:05.062443 5870 leader_election.cc:302] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 916887cd133a48868c19fd3356296ff3; no voters:
I20260926 01:05:05.062636 5870 leader_election.cc:288] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:05.062670 5875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:05.062805 5875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [term 1 LEADER]: Becoming Leader. State: Replica: 916887cd133a48868c19fd3356296ff3, State: Running, Role: LEADER
I20260926 01:05:05.062894 5870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:05.062903 5875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916887cd133a48868c19fd3356296ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } }
I20260926 01:05:05.063279 5876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "916887cd133a48868c19fd3356296ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916887cd133a48868c19fd3356296ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } }
I20260926 01:05:05.063357 5876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:05.063536 5877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 916887cd133a48868c19fd3356296ff3. Latest consensus state: current_term: 1 leader_uuid: "916887cd133a48868c19fd3356296ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916887cd133a48868c19fd3356296ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } }
I20260926 01:05:05.063599 5877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:05.063728 5882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:05.064442 5882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:05.067453 5882 catalog_manager.cc:1347] Generated new cluster ID: 09dd36992b0f4f1b9d0107853eaf444b
I20260926 01:05:05.067517 5882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:05.068269 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:05.062338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.0.98.62:36457)
0926 01:05:05.062654 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:05.062663 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:05.062697 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:05:05.062818 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:05.062825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:05.062898 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:05.063276 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:05.063380 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:05.065208 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:05.066158 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:05.066803 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:05.066942 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:05.067639 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:05.067643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:05.067644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:05.067812 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:05.067845 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:05.067857 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:05.067910 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:05.067993 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 01:05:05.068072 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":150,"threads_started":1}
I20260926 01:05:05.069090 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:05:05.081972 5882 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:05.082625 5882 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:05.095835 5882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 916887cd133a48868c19fd3356296ff3: Generated new TSK 0
I20260926 01:05:05.096052 5882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:05:05.186725 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:05.103943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.0.98.62:36457)
0926 01:05:05.103989 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:05.103991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:05.104104 (+ 113us) server_negotiation.cc:407] Connection header received
0926 01:05:05.104196 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:05.104197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:05.104208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:05.104272 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0926 01:05:05.104436 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:05.104933 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:05.106089 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:05.106308 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:05.106374 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:05.106619 (+ 245us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 01:05:05.186484 (+ 79865us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 01:05:05.186661 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33190: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 01:05:05.186838 5874 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33190: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 01:05:05.186885 5899 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:05.103889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36457 (local address 127.0.0.1:33190)
0926 01:05:05.104041 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:05.104052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:05.104118 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:05.104296 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:05.104297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:05.104300 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 01:05:05.104369 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:05.104372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:05.105012 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:05.105013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:05.105963 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:05.105966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:05.106419 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:05.106429 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:05.106497 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:05.106504 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 01:05:05.186654 (+ 80150us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 01:05:05.186784 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36457: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260926 01:05:05.187229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 5803
2026-09-26T01:05:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (294 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (93 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:05:05 dist-test-slave-l73v krb5kdc[5911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:05:05 dist-test-slave-l73v krb5kdc[5911](info): set up 2 sockets
Sep 26 01:05:05 dist-test-slave-l73v krb5kdc[5911](info): commencing operation
krb5kdc: starting...
W20260926 01:05:06.338678 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:05:06 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:05:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:05:06.457548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44557
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:06.537385 5924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:06.537628 5924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:06.537652 5924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:06.538877 5924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:06.538919 5924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:06.538933 5924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:06.538944 5924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:06.538954 5924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:06.540524 5924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:06.540788 5924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:06.541009 5924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:06.545838 5929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:06.545912 5924 server_base.cc:1034] running on GCE node
W20260926 01:05:06.545946 5930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:06.546018 5932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:06.546406 5924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:06.546711 5924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:06.547869 5924 hybrid_clock.cc:648] HybridClock initialized: now 1790384706547857 us; error 32 us; skew 500 ppm
Sep 26 01:05:06 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:06.550287 5924 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:05:06.550787 5924 webserver.cc:460] Webserver started at http://127.0.98.62:33713/ using document root <none> and password file <none>
I20260926 01:05:06.550983 5924 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:06.551029 5924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:06.551149 5924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:06.552197 5924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "c646b3b43e9944deab1fe7dc894308f2"
format_stamp: "Formatted at 2026-09-26 01:05:06 on dist-test-slave-l73v"
server_key: "8a745bf6b5f202fbab36b0a334b3842c"
server_key_iv: "9bee26e836c1a0334b1dcda42c901b2f"
server_key_version: "encryptionkey@0"
I20260926 01:05:06.552527 5924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "c646b3b43e9944deab1fe7dc894308f2"
format_stamp: "Formatted at 2026-09-26 01:05:06 on dist-test-slave-l73v"
server_key: "8a745bf6b5f202fbab36b0a334b3842c"
server_key_iv: "9bee26e836c1a0334b1dcda42c901b2f"
server_key_version: "encryptionkey@0"
I20260926 01:05:06.553782 5924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:05:06.554574 5939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:06.554750 5924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:06.554840 5924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "c646b3b43e9944deab1fe7dc894308f2"
format_stamp: "Formatted at 2026-09-26 01:05:06 on dist-test-slave-l73v"
server_key: "8a745bf6b5f202fbab36b0a334b3842c"
server_key_iv: "9bee26e836c1a0334b1dcda42c901b2f"
server_key_version: "encryptionkey@0"
I20260926 01:05:06.554903 5924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:06.566144 5924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:06.568266 5924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:06.568406 5924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:06.572216 5924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44557
I20260926 01:05:06.572255 5991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44557 every 8 connection(s)
I20260926 01:05:06.572743 5924 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:06.573365 5992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:06.575804 5992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2: Bootstrap starting.
I20260926 01:05:06.576381 5992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:06.576651 5992 log.cc:824] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:06.577409 5992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2: No bootstrap required, opened a new log
I20260926 01:05:06.579092 5992 raft_consensus.cc:348] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c646b3b43e9944deab1fe7dc894308f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44557 } }
I20260926 01:05:06.579244 5992 raft_consensus.cc:374] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:06.579276 5992 raft_consensus.cc:729] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c646b3b43e9944deab1fe7dc894308f2, State: Initialized, Role: FOLLOWER
I20260926 01:05:06.579416 5992 consensus_queue.cc:260] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c646b3b43e9944deab1fe7dc894308f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44557 } }
I20260926 01:05:06.579489 5992 raft_consensus.cc:388] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:06.579519 5992 raft_consensus.cc:482] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:06.579566 5992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:06.580497 5992 raft_consensus.cc:504] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c646b3b43e9944deab1fe7dc894308f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44557 } }
I20260926 01:05:06.580606 5992 leader_election.cc:302] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c646b3b43e9944deab1fe7dc894308f2; no voters:
I20260926 01:05:06.580780 5992 leader_election.cc:288] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:06.580850 5995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:06.581007 5995 raft_consensus.cc:686] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [term 1 LEADER]: Becoming Leader. State: Replica: c646b3b43e9944deab1fe7dc894308f2, State: Running, Role: LEADER
I20260926 01:05:06.581023 5992 sys_catalog.cc:564] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:06.581142 5995 consensus_queue.cc:237] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c646b3b43e9944deab1fe7dc894308f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44557 } }
I20260926 01:05:06.581511 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c646b3b43e9944deab1fe7dc894308f2. Latest consensus state: current_term: 1 leader_uuid: "c646b3b43e9944deab1fe7dc894308f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c646b3b43e9944deab1fe7dc894308f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44557 } } }
I20260926 01:05:06.581585 5996 sys_catalog.cc:455] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c646b3b43e9944deab1fe7dc894308f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c646b3b43e9944deab1fe7dc894308f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44557 } } }
I20260926 01:05:06.581674 5996 sys_catalog.cc:458] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:06.581853 6004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:06.581858 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:06.582441 6004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:06.582449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 5924
I20260926 01:05:06.582544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:06.582731 392 external_mini_cluster.cc:1428] Setting key a05e71dc9fd828d1811c9a891e99ae06
I20260926 01:05:06.584059 6004 catalog_manager.cc:1347] Generated new cluster ID: 1e5cf85e7b7e43f68aca77fbe26fa4cd
I20260926 01:05:06.584102 6004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:05:06 dist-test-slave-l73v krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:06.594149 6004 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:06.594746 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:06.583155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.0.98.62:44557)
0926 01:05:06.583363 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:06.583373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:06.583405 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:05:06.583528 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:06.583536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:06.583596 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:06.583738 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:06.584462 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:06.585860 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:06.587019 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:06.587698 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:06.587801 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:06.590734 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:06.590739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:06.590740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:06.590753 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:06.592729 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:06.593175 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:06.593179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:06.593180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:06.593299 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:06.593489 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:06.593490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:06.593491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:06.593747 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:06.594302 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:06.594372 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 01:05:06.594560 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260926 01:05:06.594926 6004 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:06.599882 6004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c646b3b43e9944deab1fe7dc894308f2: Generated new TSK 0
I20260926 01:05:06.600093 6004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:05:06.629849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44557
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:06.712709 6018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:06.712953 6018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:06.712989 6018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:06.714323 6018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:06.714409 6018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:06.714455 6018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:06.716248 6018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:06.716513 6018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:06.716754 6018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:06.722260 6024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:06.722260 6023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:06.722361 6018 server_base.cc:1034] running on GCE node
W20260926 01:05:06.722260 6026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:06.722777 6018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:06.723031 6018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:06.724201 6018 hybrid_clock.cc:648] HybridClock initialized: now 1790384706724185 us; error 32 us; skew 500 ppm
Sep 26 01:05:06 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:06.727015 6018 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:05:06.727491 6018 webserver.cc:460] Webserver started at http://127.0.98.1:44385/ using document root <none> and password file <none>
I20260926 01:05:06.727723 6018 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:06.727775 6018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:06.727911 6018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:06.729034 6018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "ff5dfabf603644c79bc19b61a7c9ef86"
format_stamp: "Formatted at 2026-09-26 01:05:06 on dist-test-slave-l73v"
server_key: "abb057fc76959715041b5ffa695ed368"
server_key_iv: "37f19570f465ede27bcb8d79de824148"
server_key_version: "encryptionkey@0"
I20260926 01:05:06.729373 6018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ff5dfabf603644c79bc19b61a7c9ef86"
format_stamp: "Formatted at 2026-09-26 01:05:06 on dist-test-slave-l73v"
server_key: "abb057fc76959715041b5ffa695ed368"
server_key_iv: "37f19570f465ede27bcb8d79de824148"
server_key_version: "encryptionkey@0"
I20260926 01:05:06.730762 6018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:05:06.731509 6033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:06.731664 6018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:05:06.731741 6018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "ff5dfabf603644c79bc19b61a7c9ef86"
format_stamp: "Formatted at 2026-09-26 01:05:06 on dist-test-slave-l73v"
server_key: "abb057fc76959715041b5ffa695ed368"
server_key_iv: "37f19570f465ede27bcb8d79de824148"
server_key_version: "encryptionkey@0"
I20260926 01:05:06.731812 6018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:06.744002 6018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:06.746083 6018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:06.746259 6018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:06.746516 6018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:06.746800 6018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:06.746830 6018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:06.746852 6018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:06.746868 6018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:06.753322 6018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33033
I20260926 01:05:06.753387 6146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33033 every 8 connection(s)
I20260926 01:05:06.753907 6018 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:05:06.760299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 6018
I20260926 01:05:06.760416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:05:06.760625 392 external_mini_cluster.cc:1428] Setting key 819a7dd65cbfbd3f2e3175d04374f942
Sep 26 01:05:06 dist-test-slave-l73v krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:06.770056 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:06.759939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46259 (local address 127.0.98.62:44557)
0926 01:05:06.759994 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:06.759997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:06.760335 (+ 338us) server_negotiation.cc:407] Connection header received
0926 01:05:06.761466 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:06.761467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:06.761479 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:06.761522 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:06.762183 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:06.762892 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:06.764316 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:06.764544 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:06.764618 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:06.767138 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:06.767140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:06.767141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:06.767149 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:06.768536 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:06.769059 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:06.769060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:06.769060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:06.769110 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:06.769393 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:06.769394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:06.769394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:06.769501 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:06.769544 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:06.769907 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 01:05:06.769983 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:05:06.770339 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:06.754793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44557 (local address 127.0.98.1:46259)
0926 01:05:06.760186 (+ 5393us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:06.760233 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:06.761364 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:06.761608 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:06.761612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:06.761894 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:06.762089 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:06.762093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:06.762996 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:06.762997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:06.764202 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:06.764206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:06.764636 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:06.764637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:06.764798 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:06.765387 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:06.765395 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:06.767015 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:06.768630 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:06.768639 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:06.768643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:06.768978 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:06.769173 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:06.769174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:06.769175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:06.769326 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:06.769559 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:06.769565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:06.769747 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 01:05:06.769885 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 01:05:06.770079 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5347,"thread_start_us":90,"threads_started":1}
I20260926 01:05:06.770923 6147 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44557
I20260926 01:05:06.771031 6147 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:06.771216 6147 heartbeater.cc:511] Master 127.0.98.62:44557 requested a full tablet report, sending...
I20260926 01:05:06.771775 5956 ts_manager.cc:194] Registered new tserver with Master: ff5dfabf603644c79bc19b61a7c9ef86 (127.0.98.1:33033)
I20260926 01:05:06.772428 5956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:05:06.790231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44557
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:06.873950 6152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:06.874222 6152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:06.874264 6152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:06.875574 6152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:06.875638 6152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:06.875680 6152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:06.877357 6152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:06.877645 6152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:06.877895 6152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:06.883306 6160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:06.883288 6157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:06.883323 6152 server_base.cc:1034] running on GCE node
W20260926 01:05:06.883288 6158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:06.883687 6152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:06.883906 6152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:06.885040 6152 hybrid_clock.cc:648] HybridClock initialized: now 1790384706885026 us; error 33 us; skew 500 ppm
Sep 26 01:05:06 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:06.887750 6152 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:05:06.888269 6152 webserver.cc:460] Webserver started at http://127.0.98.2:37171/ using document root <none> and password file <none>
I20260926 01:05:06.888499 6152 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:06.888554 6152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:06.888689 6152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:06.889763 6152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "9245e74a9e414231b239f63be96fedd1"
format_stamp: "Formatted at 2026-09-26 01:05:06 on dist-test-slave-l73v"
server_key: "2369c42eb6da69d86c5d99c05fb4e129"
server_key_iv: "e88f8f9db8197b0ccaf9ce180d0eb0ac"
server_key_version: "encryptionkey@0"
I20260926 01:05:06.890125 6152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9245e74a9e414231b239f63be96fedd1"
format_stamp: "Formatted at 2026-09-26 01:05:06 on dist-test-slave-l73v"
server_key: "2369c42eb6da69d86c5d99c05fb4e129"
server_key_iv: "e88f8f9db8197b0ccaf9ce180d0eb0ac"
server_key_version: "encryptionkey@0"
I20260926 01:05:06.891601 6152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:06.892397 6167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:06.892587 6152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:06.892695 6152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "9245e74a9e414231b239f63be96fedd1"
format_stamp: "Formatted at 2026-09-26 01:05:06 on dist-test-slave-l73v"
server_key: "2369c42eb6da69d86c5d99c05fb4e129"
server_key_iv: "e88f8f9db8197b0ccaf9ce180d0eb0ac"
server_key_version: "encryptionkey@0"
I20260926 01:05:06.892769 6152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:06.916280 6152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:06.918367 6152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:06.918540 6152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:06.918773 6152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:06.919070 6152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:06.919103 6152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:06.919133 6152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:06.919152 6152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:06.924686 6152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33843
I20260926 01:05:06.924737 6280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33843 every 8 connection(s)
I20260926 01:05:06.925067 6152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:05:06.925493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 6152
I20260926 01:05:06.925580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:05:06.925782 392 external_mini_cluster.cc:1428] Setting key 0943ee049cf043f24677b3ea759ecb03
Sep 26 01:05:06 dist-test-slave-l73v krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:05:06.935364 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:06.925938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33051 (local address 127.0.98.62:44557)
0926 01:05:06.925989 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:06.925991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:06.926300 (+ 309us) server_negotiation.cc:407] Connection header received
0926 01:05:06.927373 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:06.927374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:06.927390 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:06.927455 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:06.928151 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:06.928659 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:06.929741 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:06.929936 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:06.930002 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:06.932336 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:06.932337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:06.932338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:06.932347 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:06.933701 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:06.934257 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:06.934258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:06.934259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:06.934318 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:06.934586 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:06.934587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:06.934587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:06.934778 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:06.934833 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:06.935174 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 01:05:06.935260 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:05:06.935464 6283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:06.925901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44557 (local address 127.0.98.2:33051)
0926 01:05:06.926174 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:06.926216 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:06.927254 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:06.927532 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:06.927537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:06.927833 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:06.928053 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:06.928058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:06.928753 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:06.928754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:06.929651 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:06.929654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:06.930024 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:06.930025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:06.930152 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:06.930708 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:06.930717 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:06.932141 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:06.933838 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:06.933855 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:06.933858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:06.934163 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:06.934412 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:06.934412 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:06.934413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:06.934522 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:06.934837 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:06.934843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:06.935000 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 01:05:06.935124 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 01:05:06.935270 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 01:05:06.935954 6281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44557
I20260926 01:05:06.936049 6281 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:06.936231 6281 heartbeater.cc:511] Master 127.0.98.62:44557 requested a full tablet report, sending...
I20260926 01:05:06.936615 5956 ts_manager.cc:194] Registered new tserver with Master: 9245e74a9e414231b239f63be96fedd1 (127.0.98.2:33843)
I20260926 01:05:06.937099 5956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:05:06.951391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44557
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:07.032579 6286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:07.032814 6286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:07.032837 6286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:07.034122 6286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:07.034176 6286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:07.034204 6286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:07.035948 6286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:07.036185 6286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:07.036408 6286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:07.041348 6291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:07.041469 6294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:07.041495 6292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:07.041530 6286 server_base.cc:1034] running on GCE node
I20260926 01:05:07.041795 6286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:07.042037 6286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:07.043169 6286 hybrid_clock.cc:648] HybridClock initialized: now 1790384707043151 us; error 33 us; skew 500 ppm
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:07.045542 6286 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:05:07.046017 6286 webserver.cc:460] Webserver started at http://127.0.98.3:33879/ using document root <none> and password file <none>
I20260926 01:05:07.046258 6286 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:07.046309 6286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:07.046456 6286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:07.047525 6286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "f357540210664a569206c558aa4b2d60"
format_stamp: "Formatted at 2026-09-26 01:05:07 on dist-test-slave-l73v"
server_key: "4d6f61f3592b3d200f854731620d0b98"
server_key_iv: "9e61df14518d902cf0eda8dc40281b23"
server_key_version: "encryptionkey@0"
I20260926 01:05:07.047854 6286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f357540210664a569206c558aa4b2d60"
format_stamp: "Formatted at 2026-09-26 01:05:07 on dist-test-slave-l73v"
server_key: "4d6f61f3592b3d200f854731620d0b98"
server_key_iv: "9e61df14518d902cf0eda8dc40281b23"
server_key_version: "encryptionkey@0"
I20260926 01:05:07.049116 6286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:05:07.049830 6301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:07.049988 6286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:05:07.050086 6286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "f357540210664a569206c558aa4b2d60"
format_stamp: "Formatted at 2026-09-26 01:05:07 on dist-test-slave-l73v"
server_key: "4d6f61f3592b3d200f854731620d0b98"
server_key_iv: "9e61df14518d902cf0eda8dc40281b23"
server_key_version: "encryptionkey@0"
I20260926 01:05:07.050149 6286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:07.062837 6286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:07.064996 6286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:07.065143 6286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:07.065366 6286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:07.065657 6286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:07.065689 6286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:07.065721 6286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:07.065744 6286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:07.071487 6286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42955
I20260926 01:05:07.071554 6414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42955 every 8 connection(s)
I20260926 01:05:07.071848 6286 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:05:07.076304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 6286
I20260926 01:05:07.076407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:05:07.076599 392 external_mini_cluster.cc:1428] Setting key 67454bd97301170a25af6d1b482721b2
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:07.081869 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:07.072603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40261 (local address 127.0.98.62:44557)
0926 01:05:07.072675 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:07.072677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:07.072790 (+ 113us) server_negotiation.cc:407] Connection header received
0926 01:05:07.073733 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:07.073734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:07.073745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:07.073800 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:07.074469 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:07.075062 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:07.076118 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:07.076315 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:07.076381 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:07.078911 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:07.078912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:07.078912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:07.078919 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:07.080298 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:07.080756 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:07.080757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:07.080757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:07.080812 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:07.081206 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:07.081207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:07.081207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:07.081336 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:07.081401 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:07.081725 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 01:05:07.081801 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 01:05:07.082144 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:07.072521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44557 (local address 127.0.98.3:40261)
0926 01:05:07.072675 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:07.072713 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:07.073625 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:07.073859 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:07.073863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:07.074137 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:07.074343 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:07.074348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:07.075140 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:07.075141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:07.076005 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:07.076007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:07.076485 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:07.076487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:07.076650 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:07.077208 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:07.077218 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:07.078801 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:07.080388 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:07.080394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:07.080396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:07.080679 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:07.080950 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:07.080951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:07.080952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:07.081118 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:07.081408 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:07.081414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:07.081565 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 01:05:07.081687 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 01:05:07.081843 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260926 01:05:07.082609 6415 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44557
I20260926 01:05:07.082705 6415 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:07.082854 6415 heartbeater.cc:511] Master 127.0.98.62:44557 requested a full tablet report, sending...
I20260926 01:05:07.083395 5956 ts_manager.cc:194] Registered new tserver with Master: f357540210664a569206c558aa4b2d60 (127.0.98.3:42955)
I20260926 01:05:07.083884 5956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40261
I20260926 01:05:07.088511 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/test-admin.keytab.
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:07.104233 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 01:05:07.106125 6422 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:07.105230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.0.98.1:33033)
0926 01:05:07.105420 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:07.105425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:07.105437 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:07.105462 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:07.105464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:07.105491 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:07.105561 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:07.106040 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38096: BlockingRecv error: recv got EOF from 127.0.0.1:38096 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
W20260926 01:05:07.106245 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:07.107658 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:07.108942 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:05:07.117408 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:07.109700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.0.98.1:33033)
0926 01:05:07.109761 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:07.109763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:07.109822 (+ 59us) server_negotiation.cc:407] Connection header received
0926 01:05:07.109873 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:07.109874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:07.109886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:07.109936 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:07.110362 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:07.111190 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:07.111958 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:07.112349 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:07.112412 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:07.114712 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:07.114724 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:07.114724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:07.114732 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:07.116289 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:07.116642 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:07.116645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:07.116646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:07.116700 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:07.116865 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:07.116866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:07.116866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:07.117012 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:07.117172 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:07.117214 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 01:05:07.117299 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 01:05:07.119310 6422 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:07.118780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.0.98.1:33033)
0926 01:05:07.118816 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:07.118818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:07.118863 (+ 45us) server_negotiation.cc:407] Connection header received
0926 01:05:07.118933 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:07.118934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:07.118945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:07.118987 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:07.119271 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38122: BlockingRecv error: recv got EOF from 127.0.0.1:38122 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 01:05:07.119530 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5cc)
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:07.120565 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:07.121515 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:05:07.128875 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:07.122130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.0.98.1:33033)
0926 01:05:07.122234 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:07.122236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:07.122245 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:07.122265 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:07.122266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:07.122274 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:07.122327 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:07.122700 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:07.123205 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:07.123898 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:07.124116 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:07.124174 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:07.126252 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:07.126254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:07.126254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:07.126262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:07.127601 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:07.127994 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:07.127996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:07.127996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:07.128052 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:07.128255 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:07.128256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:07.128256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:07.128381 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:07.128470 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:07.128615 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 01:05:07.128692 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 01:05:07.130606 6422 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:07.129966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.0.98.1:33033)
0926 01:05:07.130009 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:07.130011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:07.130088 (+ 77us) server_negotiation.cc:407] Connection header received
0926 01:05:07.130153 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:07.130154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:07.130162 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:07.130206 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:07.130549 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38150: BlockingRecv error: recv got EOF from 127.0.0.1:38150 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 01:05:07.130707 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790384680280236-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:07.131774 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:07.132778 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[5911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:05:07.141024 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:07.133399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.0.98.1:33033)
0926 01:05:07.133452 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:07.133454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:07.133461 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:05:07.133524 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:07.133525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:07.133532 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:07.133569 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:07.134186 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:07.134828 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:07.136049 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:07.136325 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:07.136392 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:07.138747 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:07.138749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:07.138749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:07.138758 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:07.140044 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:07.140366 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:07.140366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:07.140367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:07.140435 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:07.140571 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:07.140572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:07.140572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:07.140673 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:07.140746 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:07.140875 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 01:05:07.140934 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:05:07.141767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 6018
I20260926 01:05:07.145970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 6152
I20260926 01:05:07.150045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 6286
I20260926 01:05:07.153923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 5924
2026-09-26T01:05:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1783 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[6435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[6435](info): set up 2 sockets
Sep 26 01:05:07 dist-test-slave-l73v krb5kdc[6435](info): commencing operation
krb5kdc: starting...
W20260926 01:05:08.119189 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:05:08 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:05:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 01:05:08.245641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:08.340615 6448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:08.340850 6448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:08.340875 6448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:08.342177 6448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:08.342231 6448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:08.342245 6448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:08.342257 6448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:08.342269 6448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:08.344178 6448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:08.344468 6448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:08.344699 6448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:08.349962 6456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:08.350555 6448 server_base.cc:1034] running on GCE node
W20260926 01:05:08.350795 6453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:08.350800 6454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:08.351263 6448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:08.351622 6448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:08.352846 6448 hybrid_clock.cc:648] HybridClock initialized: now 1790384708352825 us; error 38 us; skew 500 ppm
Sep 26 01:05:08 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:08.359524 6448 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 01:05:08.360121 6448 webserver.cc:460] Webserver started at http://127.0.98.62:33289/ using document root <none> and password file <none>
I20260926 01:05:08.360344 6448 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:08.360396 6448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:08.360543 6448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:08.361855 6448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "b0573a39c6e042d19458b0849f990d9b"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "17f5ba693c42deeda7325386fcac5cfa"
server_key_iv: "2859ba4ddf004c18f1919df147b4ff9c"
server_key_version: "encryptionkey@0"
I20260926 01:05:08.362237 6448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0573a39c6e042d19458b0849f990d9b"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "17f5ba693c42deeda7325386fcac5cfa"
server_key_iv: "2859ba4ddf004c18f1919df147b4ff9c"
server_key_version: "encryptionkey@0"
I20260926 01:05:08.363745 6448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:08.364765 6463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:08.365111 6448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:08.365221 6448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "b0573a39c6e042d19458b0849f990d9b"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "17f5ba693c42deeda7325386fcac5cfa"
server_key_iv: "2859ba4ddf004c18f1919df147b4ff9c"
server_key_version: "encryptionkey@0"
I20260926 01:05:08.365298 6448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:08.380695 6448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:08.383193 6448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:08.383374 6448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:08.388851 6448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35137
I20260926 01:05:08.388945 6515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35137 every 8 connection(s)
I20260926 01:05:08.389295 6448 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:08.390327 6516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:08.391393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 6448
I20260926 01:05:08.391526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:08.391716 392 external_mini_cluster.cc:1428] Setting key 3ddf90431668f4c78d1879acd68676d0
I20260926 01:05:08.393055 6516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b: Bootstrap starting.
I20260926 01:05:08.393685 6516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:08.393970 6516 log.cc:824] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:08.394809 6516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b: No bootstrap required, opened a new log
I20260926 01:05:08.397010 6516 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } }
I20260926 01:05:08.397217 6516 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:08.397253 6516 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0573a39c6e042d19458b0849f990d9b, State: Initialized, Role: FOLLOWER
I20260926 01:05:08.397413 6516 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } }
I20260926 01:05:08.397511 6516 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:08.397554 6516 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:08.397605 6516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 0 FOLLOWER]: Advancing to term 1
Sep 26 01:05:08 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:05:08.405527 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:08.392285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.0.98.62:35137)
0926 01:05:08.392486 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:08.392497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:08.392552 (+ 55us) server_negotiation.cc:407] Connection header received
0926 01:05:08.393020 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:08.393028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:08.393090 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:08.393238 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:08.393969 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:08.395360 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:08.396550 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:08.397230 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:08.397779 (+ 549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:08.400776 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:08.400782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:08.400782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:08.400797 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:08.402763 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:08.403477 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:08.403484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:08.403484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:08.403691 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:08.404073 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:08.404075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:08.404075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:08.404477 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:08.404760 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:08.405096 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 01:05:08.405263 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260926 01:05:08.468511 6516 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } }
I20260926 01:05:08.468755 6516 leader_election.cc:302] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0573a39c6e042d19458b0849f990d9b; no voters:
I20260926 01:05:08.469000 6516 leader_election.cc:288] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:08.469050 6521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:08.469213 6521 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 1 LEADER]: Becoming Leader. State: Replica: b0573a39c6e042d19458b0849f990d9b, State: Running, Role: LEADER
I20260926 01:05:08.469303 6521 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } }
I20260926 01:05:08.469378 6516 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:08.469650 6521 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } }
I20260926 01:05:08.469722 6521 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:08.469834 6521 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0573a39c6e042d19458b0849f990d9b. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } }
I20260926 01:05:08.469904 6521 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:08.470034 6525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:08.470702 6525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:08.472683 6525 catalog_manager.cc:1347] Generated new cluster ID: e6ab4cb480ae405ab602f8b999ba1c19
I20260926 01:05:08.472769 6525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:08.480166 6525 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:08.481024 6525 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:08.485846 6525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b: Generated new TSK 0
I20260926 01:05:08.486053 6525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 01:05:08.587288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35137
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:08.670660 6542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:08.671003 6542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:08.671056 6542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:08.672739 6542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:08.672809 6542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:08.672845 6542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:08.674690 6542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:08.674932 6542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:08.675148 6542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:08.680671 6547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:08.680671 6548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:08.680671 6550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:08.681283 6542 server_base.cc:1034] running on GCE node
I20260926 01:05:08.681530 6542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:08.681833 6542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:08.683005 6542 hybrid_clock.cc:648] HybridClock initialized: now 1790384708682992 us; error 34 us; skew 500 ppm
Sep 26 01:05:08 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:08.685631 6542 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 01:05:08.686128 6542 webserver.cc:460] Webserver started at http://127.0.98.1:37287/ using document root <none> and password file <none>
I20260926 01:05:08.686333 6542 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:08.686409 6542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:08.686548 6542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:08.687654 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "478ce1ac7e994e7db5035b881a7bfa00"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "287d0f58cf43b0d28abfc4ac8f453eb9"
server_key_iv: "464c72a58a4175f7e4c3d08252b5a655"
server_key_version: "encryptionkey@0"
I20260926 01:05:08.687995 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "478ce1ac7e994e7db5035b881a7bfa00"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "287d0f58cf43b0d28abfc4ac8f453eb9"
server_key_iv: "464c72a58a4175f7e4c3d08252b5a655"
server_key_version: "encryptionkey@0"
I20260926 01:05:08.689369 6542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:08.690409 6557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:08.690593 6542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:08.690683 6542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "478ce1ac7e994e7db5035b881a7bfa00"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "287d0f58cf43b0d28abfc4ac8f453eb9"
server_key_iv: "464c72a58a4175f7e4c3d08252b5a655"
server_key_version: "encryptionkey@0"
I20260926 01:05:08.690760 6542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:08.703399 6542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:08.705681 6542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:08.705827 6542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:08.706084 6542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:08.706490 6542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:08.706535 6542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:08.706558 6542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:08.706569 6542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:08.712432 6542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43003
I20260926 01:05:08.712488 6670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43003 every 8 connection(s)
I20260926 01:05:08.712814 6542 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:05:08 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:05:08.722612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 6542
I20260926 01:05:08.722725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:05:08.722926 392 external_mini_cluster.cc:1428] Setting key 02572572e5699af8a095ee86a56f1493
I20260926 01:05:08.723290 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:08.713561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34741 (local address 127.0.98.62:35137)
0926 01:05:08.713630 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:08.713632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:08.713870 (+ 238us) server_negotiation.cc:407] Connection header received
0926 01:05:08.714461 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:08.714463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:08.714478 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:08.714532 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:08.715564 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:08.716110 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:08.717210 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:08.717439 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:08.717505 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:08.720018 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:08.720022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:08.720022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:08.720036 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:08.721556 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:08.722019 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:08.722019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:08.722020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:08.722075 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:08.722513 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:08.722514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:08.722515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:08.722679 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:08.722757 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:08.723120 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 01:05:08.723205 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:05:08.723526 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:08.713528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:34741)
0926 01:05:08.713706 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:08.713777 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:08.714319 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:08.714603 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:08.714606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:08.715220 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:08.715434 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:08.715439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:08.716214 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:08.716215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:08.717108 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:08.717110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:08.717523 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:08.717524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:08.717652 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:08.718266 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:08.718275 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:08.719826 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:08.721667 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:08.721675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:08.721678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:08.721942 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:08.722156 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:08.722157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:08.722158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:08.722336 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:08.722772 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:08.722780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:08.722955 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 01:05:08.723071 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 01:05:08.723271 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260926 01:05:08.724048 6671 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35137
I20260926 01:05:08.724376 6671 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:08.724751 6671 heartbeater.cc:511] Master 127.0.98.62:35137 requested a full tablet report, sending...
I20260926 01:05:08.725453 6480 ts_manager.cc:194] Registered new tserver with Master: 478ce1ac7e994e7db5035b881a7bfa00 (127.0.98.1:43003)
I20260926 01:05:08.726097 6480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34741
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 01:05:08.753758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35137
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:08.840489 6676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:08.840729 6676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:08.840755 6676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:08.841982 6676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:08.842026 6676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:08.842053 6676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:08.843613 6676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:08.843843 6676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:08.844053 6676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:08.849709 6682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:08.849730 6681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:08.849829 6676 server_base.cc:1034] running on GCE node
W20260926 01:05:08.849738 6684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:08.850211 6676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:08.850564 6676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:08.851723 6676 hybrid_clock.cc:648] HybridClock initialized: now 1790384708851706 us; error 32 us; skew 500 ppm
Sep 26 01:05:08 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:08.854457 6676 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 01:05:08.854957 6676 webserver.cc:460] Webserver started at http://127.0.98.2:35757/ using document root <none> and password file <none>
I20260926 01:05:08.855163 6676 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:08.855208 6676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:08.855325 6676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:08.856431 6676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "32e08a68241a46fb9685683facdbaf77"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "0b9c88e05de18d6b2b343911070a2385"
server_key_iv: "121278c8af925e131cd74cec840ab8e8"
server_key_version: "encryptionkey@0"
I20260926 01:05:08.856774 6676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "32e08a68241a46fb9685683facdbaf77"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "0b9c88e05de18d6b2b343911070a2385"
server_key_iv: "121278c8af925e131cd74cec840ab8e8"
server_key_version: "encryptionkey@0"
I20260926 01:05:08.858152 6676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:05:08.859105 6691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:08.859426 6676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:08.859529 6676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "32e08a68241a46fb9685683facdbaf77"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "0b9c88e05de18d6b2b343911070a2385"
server_key_iv: "121278c8af925e131cd74cec840ab8e8"
server_key_version: "encryptionkey@0"
I20260926 01:05:08.859620 6676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:08.868595 6676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:08.870921 6676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:08.871047 6676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:08.871289 6676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:08.871589 6676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:08.871621 6676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:08.871642 6676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:08.871656 6676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:08.877413 6676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39833
I20260926 01:05:08.877458 6804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39833 every 8 connection(s)
I20260926 01:05:08.877805 6676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:05:08.879596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 6676
I20260926 01:05:08.879801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:05:08.880137 392 external_mini_cluster.cc:1428] Setting key 21b6a2ca77cba741011e133b2d2009af
Sep 26 01:05:08 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:05:08.889498 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:08.878851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32799 (local address 127.0.98.62:35137)
0926 01:05:08.878923 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:08.878925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:08.879103 (+ 178us) server_negotiation.cc:407] Connection header received
0926 01:05:08.879660 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:08.879661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:08.879675 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:08.879734 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:08.880866 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:08.881356 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:08.882655 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:08.882859 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:08.882918 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:08.885730 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:08.885732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:08.885732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:08.885740 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:08.887799 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:08.888421 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:08.888422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:08.888423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:08.888491 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:08.888804 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:08.888806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:08.888806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:08.888954 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:08.889018 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:08.889334 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 01:05:08.889429 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:05:08.889596 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:08.878821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:32799)
0926 01:05:08.878981 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:08.879021 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:08.879553 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:08.879817 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:08.879821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:08.880562 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:08.880775 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:08.880781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:08.881459 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:08.881460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:08.882536 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:08.882540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:08.882949 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:08.882950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:08.883096 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:08.883666 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:08.883676 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:08.885599 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:08.887932 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:08.887945 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:08.887950 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:08.888316 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:08.888573 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:08.888574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:08.888574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:08.888739 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:08.889025 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:08.889031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:08.889173 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 01:05:08.889268 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 01:05:08.889409 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260926 01:05:08.890146 6805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35137
I20260926 01:05:08.890245 6805 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:08.890465 6805 heartbeater.cc:511] Master 127.0.98.62:35137 requested a full tablet report, sending...
I20260926 01:05:08.890829 6480 ts_manager.cc:194] Registered new tserver with Master: 32e08a68241a46fb9685683facdbaf77 (127.0.98.2:39833)
I20260926 01:05:08.891249 6480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 01:05:08.915328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35137
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:09.001125 6810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:09.001364 6810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:09.001390 6810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:09.002875 6810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:09.003017 6810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:09.003054 6810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:09.004828 6810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:09.005286 6810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:09.005580 6810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:09.011062 6815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.011062 6810 server_base.cc:1034] running on GCE node
W20260926 01:05:09.011072 6816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.011076 6818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.011494 6810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:09.011749 6810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:09.012929 6810 hybrid_clock.cc:648] HybridClock initialized: now 1790384709012918 us; error 30 us; skew 500 ppm
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:09.015785 6810 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 01:05:09.016319 6810 webserver.cc:460] Webserver started at http://127.0.98.3:46705/ using document root <none> and password file <none>
I20260926 01:05:09.016547 6810 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:09.016592 6810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:09.016705 6810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:09.017941 6810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "edc5dc1a23274a64aa4b90e7f583dbeb"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "2f9e497f743ecdd6d3610f608f6159e9"
server_key_iv: "4c2bf730e6ca025be015f3722bbb41a6"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.018304 6810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "edc5dc1a23274a64aa4b90e7f583dbeb"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "2f9e497f743ecdd6d3610f608f6159e9"
server_key_iv: "4c2bf730e6ca025be015f3722bbb41a6"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.019716 6810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:09.020560 6825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.020717 6810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:05:09.020798 6810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "edc5dc1a23274a64aa4b90e7f583dbeb"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "2f9e497f743ecdd6d3610f608f6159e9"
server_key_iv: "4c2bf730e6ca025be015f3722bbb41a6"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.020869 6810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:09.027647 6810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:09.030047 6810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:09.030190 6810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:09.030848 6810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:09.031219 6810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:09.031256 6810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.031280 6810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:09.031292 6810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.037036 6810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40797
I20260926 01:05:09.037084 6938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40797 every 8 connection(s)
I20260926 01:05:09.037396 6810 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:05:09.040515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 6810
I20260926 01:05:09.040625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:05:09.040807 392 external_mini_cluster.cc:1428] Setting key 05b463555e14e7fcf94b254aa54b73c3
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:05:09.047616 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.038159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37639 (local address 127.0.98.62:35137)
0926 01:05:09.038222 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.038224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.038430 (+ 206us) server_negotiation.cc:407] Connection header received
0926 01:05:09.038914 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.038915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.038927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.038994 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.039922 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.040538 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.041907 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.042107 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.042173 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.044616 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.044617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.044617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.044624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.046014 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.046451 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.046452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.046452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.046541 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.046899 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.046900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.046901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.047044 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.047135 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.047464 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.047547 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:05:09.047771 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.038109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:37639)
0926 01:05:09.038261 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.038305 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.038831 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.039048 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.039051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.039625 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.039826 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.039830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.040649 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.040651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.041736 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.041740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.042215 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.042216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.042339 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.043062 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.043070 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.044521 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.046094 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.046100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.046102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.046321 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.046629 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.046630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.046631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.046759 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.047132 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.047139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.047288 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.047404 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.047550 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260926 01:05:09.048355 6939 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35137
I20260926 01:05:09.048481 6939 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:09.048652 6939 heartbeater.cc:511] Master 127.0.98.62:35137 requested a full tablet report, sending...
I20260926 01:05:09.048976 6480 ts_manager.cc:194] Registered new tserver with Master: edc5dc1a23274a64aa4b90e7f583dbeb (127.0.98.3:40797)
I20260926 01:05:09.049336 6480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37639
I20260926 01:05:09.052369 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 01:05:09.082893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43727
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35137,127.0.98.61:43727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:09.166203 6944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:09.166469 6944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:09.166505 6944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:09.167837 6944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:09.167892 6944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:09.167913 6944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:09.167929 6944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:09.167945 6944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:09.169700 6944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35137,127.0.98.61:43727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
W20260926 01:05:09.169844 6944 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:35137,127.0.98.61:43727'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 01:05:09.170002 6944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:09.170217 6944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:09.175254 6949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.175254 6952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.175254 6950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.175557 6944 server_base.cc:1034] running on GCE node
I20260926 01:05:09.175788 6944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:09.176080 6944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:09.177256 6944 hybrid_clock.cc:648] HybridClock initialized: now 1790384709177236 us; error 36 us; skew 500 ppm
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:09.179909 6944 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 01:05:09.180472 6944 webserver.cc:460] Webserver started at http://127.0.98.61:40219/ using document root <none> and password file <none>
I20260926 01:05:09.180684 6944 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:09.180734 6944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:09.180860 6944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:09.181914 6944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data/instance:
uuid: "d7cda31285d14786b026cc33437c40e7"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "d16602973572cc900887aa02a26c9b9f"
server_key_iv: "a9fd8579c7083027c3b50269655c5bcf"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.182264 6944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal/instance:
uuid: "d7cda31285d14786b026cc33437c40e7"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "d16602973572cc900887aa02a26c9b9f"
server_key_iv: "a9fd8579c7083027c3b50269655c5bcf"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.183684 6944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:09.184484 6959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.184657 6944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:05:09.184742 6944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal
uuid: "d7cda31285d14786b026cc33437c40e7"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "d16602973572cc900887aa02a26c9b9f"
server_key_iv: "a9fd8579c7083027c3b50269655c5bcf"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.184808 6944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:09.198876 6944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:09.201108 6944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:09.201261 6944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:09.205593 6944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43727
I20260926 01:05:09.205688 7011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43727 every 8 connection(s)
I20260926 01:05:09.206032 6944 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
I20260926 01:05:09.207060 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:09.207824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 6944
I20260926 01:05:09.209061 7012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 01:05:09.218082 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.210118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.0.98.62:35137)
0926 01:05:09.210162 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.210164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.210366 (+ 202us) server_negotiation.cc:407] Connection header received
0926 01:05:09.210414 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.210415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.210427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.210479 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.211046 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.211551 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.212437 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.212636 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.212700 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.215019 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.215021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.215021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.215031 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.216497 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.217067 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.217068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.217068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.217127 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.217394 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.217395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.217395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.217555 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.217618 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.217893 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.217988 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:05:09.218344 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.208630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.0.98.61:43727)
0926 01:05:09.208833 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.208844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.208887 (+ 43us) server_negotiation.cc:407] Connection header received
0926 01:05:09.209396 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.209406 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.209473 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.209635 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.210367 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.211271 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.212005 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.212415 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.212518 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.215387 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.215397 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.215397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.215404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.217126 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.217452 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.217455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.217455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.217505 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.217663 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.217664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.217664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.217804 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.217980 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.218072 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.218148 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260926 01:05:09.218500 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.210028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:43140)
0926 01:05:09.210221 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.210262 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.210333 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.210505 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.210508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.210847 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.210946 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.210952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.211655 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.211657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.212341 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.212345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.212712 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.212713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.212790 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.213214 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.213222 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.214791 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.216626 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.216632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.216634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.216969 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.217214 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.217215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.217215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.217329 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.217621 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.217627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.217771 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.217866 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.218035 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260926 01:05:09.219297 7012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 01:05:09.226148 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.219555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43727 (local address 127.0.0.1:59670)
0926 01:05:09.219601 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.219611 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.219651 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.219825 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.219826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.219957 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.220020 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.220022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.220793 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.220794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.221294 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.221295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.221669 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.221670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.221727 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.222187 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.222191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.223441 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.225076 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.225077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.225077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.225241 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.225428 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.225428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.225429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.225511 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.225826 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.225827 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.225874 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.225943 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.226051 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:05:09.226202 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.219648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.0.98.61:43727)
0926 01:05:09.219704 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.219706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.219713 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:05:09.219728 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.219729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.219737 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.219784 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.220109 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.220704 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.221389 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.221597 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.221655 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.223533 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.223534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.223535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.223542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.224980 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.225306 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.225307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.225307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.225356 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.225593 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.225594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.225595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.225716 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.225782 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.226012 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.226101 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":37}
I20260926 01:05:09.227556 7012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Bootstrap starting.
I20260926 01:05:09.228335 7012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:09.228559 7012 log.cc:824] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:09.229226 7012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: No bootstrap required, opened a new log
I20260926 01:05:09.230722 7012 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } }
I20260926 01:05:09.230866 7012 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:09.230886 7012 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cda31285d14786b026cc33437c40e7, State: Initialized, Role: FOLLOWER
I20260926 01:05:09.230983 7012 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } }
I20260926 01:05:09.231232 7018 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } } }
I20260926 01:05:09.231278 7012 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:09.231323 7018 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:05:09.233121 7029 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:05:09.233188 7029 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:05:09.234799 6944 master_runner.cc:428] Detected that this master d7cda31285d14786b026cc33437c40e7 is joining an existing cluster
I20260926 01:05:09.234848 6944 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43727
I20260926 01:05:09.242594 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.235959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.0.98.62:35137)
0926 01:05:09.236013 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.236015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.236112 (+ 97us) server_negotiation.cc:407] Connection header received
0926 01:05:09.236190 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.236191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.236201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.236249 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.236679 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.237416 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.238176 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.238520 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.238600 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.239690 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.239691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.239692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.239699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.241306 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.241739 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.241740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.241740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.241799 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.242008 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.242009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.242009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.242143 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.242186 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.242376 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.242494 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:05:09.242657 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.236061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.0.98.61:43727)
0926 01:05:09.236100 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.236101 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.236308 (+ 207us) server_negotiation.cc:407] Connection header received
0926 01:05:09.236372 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.236373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.236381 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.236467 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.236857 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.237526 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.238620 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.238900 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.238936 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.239964 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.239965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.239965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.239972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.241385 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.241807 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.241808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.241808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.241894 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.242104 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.242104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.242105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.242221 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.242264 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.242490 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.242568 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:05:09.243027 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.236100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43727 (local address 127.0.0.1:59672)
0926 01:05:09.236252 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.236262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.236331 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.236481 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.236482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.236719 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.236783 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.236785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.237601 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.237602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.238538 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.238541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.239001 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.239002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.239054 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.239438 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.239442 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.239861 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.241476 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.241476 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.241477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.241722 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.241982 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.241982 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.241983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.242040 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.242320 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.242321 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.242358 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.242771 (+ 413us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.242927 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260926 01:05:09.243225 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.235819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:43148)
0926 01:05:09.236047 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.236057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.236137 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.236256 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.236257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.236542 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.236613 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.236615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.237499 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.237500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.238100 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.238101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.238612 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.238613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.238685 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.239154 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.239159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.239606 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.241394 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.241395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.241395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.241657 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.241879 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.241880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.241880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.241938 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.242229 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.242230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.242270 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.242858 (+ 588us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.242993 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":28,"thread_start_us":86,"threads_started":1}
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 01:05:09.253188 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.245888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.0.98.61:43727)
0926 01:05:09.245925 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.245927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.246085 (+ 158us) server_negotiation.cc:407] Connection header received
0926 01:05:09.246177 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.246178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.246189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.246248 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.246734 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.247193 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.248063 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.248262 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.248328 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.250521 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.250523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.250523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.250530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.252026 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.252494 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.252495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.252495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.252557 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.252738 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.252739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.252739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.252834 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.252875 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.253065 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.253123 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:05:09.253265 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.245839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43727 (local address 127.0.0.1:59680)
0926 01:05:09.245982 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.246022 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.246098 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.246254 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.246258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.246602 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.246655 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.246658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.247323 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.247325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.247982 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.247984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.248331 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.248331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.248411 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.248857 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.248863 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.250446 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.252104 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.252131 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.252134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.252415 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.252617 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.252617 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.252618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.252683 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.252879 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.252884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.252943 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.253039 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.253164 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260926 01:05:09.253849 6480 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43727: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d7cda31285d14786b026cc33437c40e7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43727
}
attrs {
promote: true
}
}
dest_uuid: "b0573a39c6e042d19458b0849f990d9b"
cas_config_opid_index: -1
I20260926 01:05:09.254196 6480 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } }
I20260926 01:05:09.254709 7039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d7cda31285d14786b026cc33437c40e7 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } } }
I20260926 01:05:09.254982 6986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:09.255744 6986 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Refusing update from remote peer b0573a39c6e042d19458b0849f990d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 01:05:09.255858 7039 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43727
I20260926 01:05:09.255961 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } } }
I20260926 01:05:09.256038 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:09.256145 7039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:09.256268 6944 master.cc:537] Master@127.0.98.61:43727 shutting down...
W20260926 01:05:09.256783 6465 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 01:05:09.257356 6465 consensus_peers.cc:487] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b -> Peer d7cda31285d14786b026cc33437c40e7 (127.0.98.61:43727): Couldn't send request to peer d7cda31285d14786b026cc33437c40e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:05:09.259243 6944 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:09.259330 6944 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:09.259382 6944 tablet_replica.cc:330] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: stopping tablet replica
I20260926 01:05:09.262027 6944 master.cc:552] Master@127.0.98.61:43727 shutdown complete.
I20260926 01:05:09.262100 6944 master_runner.cc:305] Clearing existing system tablet
I20260926 01:05:09.262213 6944 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:09.262255 6944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:09.262944 6944 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.263392 7043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.263501 6944 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:09.263556 6944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.263581 6944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal
uuid: "d7cda31285d14786b026cc33437c40e7"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "d16602973572cc900887aa02a26c9b9f"
server_key_iv: "a9fd8579c7083027c3b50269655c5bcf"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.264209 6944 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:09.266040 6944 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 01:05:09.266105 6944 log.cc:1196] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 01:05:09.266436 6944 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Deleting consensus metadata
I20260926 01:05:09.266589 6944 master_runner.cc:315] Copying system tablet from 127.0.98.62:35137
I20260926 01:05:09.267210 6944 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35137
I20260926 01:05:09.274245 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.267580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.0.98.62:35137)
0926 01:05:09.267639 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.267641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.267713 (+ 72us) server_negotiation.cc:407] Connection header received
0926 01:05:09.267781 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.267781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.267791 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.267844 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.268234 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.268894 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.270016 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.270214 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.270335 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.271751 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.271752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.271752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.271759 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.273064 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.273360 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.273360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.273361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.273413 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.273725 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.273726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.273726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.273856 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.273917 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.274063 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.274145 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:05:09.274261 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.267513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:43156)
0926 01:05:09.267649 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.267660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.267730 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.267858 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.267859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.268056 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.268101 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.268103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.268976 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.268977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.269916 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.269919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.270293 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.270294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.270453 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.271360 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.271364 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.271663 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.273140 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.273142 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.273142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.273300 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.273466 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.273466 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.273467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.273631 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.273917 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.273919 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.273973 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.274032 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.274145 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260926 01:05:09.274591 6498 tablet_copy_service.cc:140] P b0573a39c6e042d19458b0849f990d9b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d7cda31285d14786b026cc33437c40e7 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43156)
I20260926 01:05:09.274676 6498 tablet_copy_service.cc:161] P b0573a39c6e042d19458b0849f990d9b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d7cda31285d14786b026cc33437c40e7 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43156: session id = d7cda31285d14786b026cc33437c40e7-00000000000000000000000000000000
I20260926 01:05:09.275277 6498 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 01:05:09.275856 6944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 01:05:09.276777 6944 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: tablet copy: Starting download of 0 data blocks...
I20260926 01:05:09.276916 6944 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: tablet copy: Starting download of 1 WAL segments...
I20260926 01:05:09.278038 6944 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 01:05:09.279095 6498 tablet_copy_service.cc:342] P b0573a39c6e042d19458b0849f990d9b: Request end of tablet copy session d7cda31285d14786b026cc33437c40e7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43156
I20260926 01:05:09.279170 6498 tablet_copy_service.cc:433] P b0573a39c6e042d19458b0849f990d9b: ending tablet copy session d7cda31285d14786b026cc33437c40e7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d7cda31285d14786b026cc33437c40e7
I20260926 01:05:09.281666 6944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:09.282860 7050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.282891 7051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.283057 6944 server_base.cc:1034] running on GCE node
W20260926 01:05:09.283056 7053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.283253 6944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:09.283386 6944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:09.284528 6944 hybrid_clock.cc:648] HybridClock initialized: now 1790384709284519 us; error 22 us; skew 500 ppm
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:09.285996 6944 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 01:05:09.286238 6944 webserver.cc:460] Webserver started at http://127.0.98.61:33011/ using document root <none> and password file <none>
I20260926 01:05:09.286342 6944 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:09.286376 6944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:09.286958 6944 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.287442 7060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.287575 6944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.287621 6944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal
uuid: "d7cda31285d14786b026cc33437c40e7"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "d16602973572cc900887aa02a26c9b9f"
server_key_iv: "a9fd8579c7083027c3b50269655c5bcf"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.287678 6944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:09.298280 6944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:09.300688 6944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:09.303725 6944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43727
I20260926 01:05:09.303920 7112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43727 every 8 connection(s)
I20260926 01:05:09.304159 6944 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
I20260926 01:05:09.305105 7113 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:05:09.305629 7113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Bootstrap starting.
I20260926 01:05:09.308612 7113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:05:09.308872 7113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Bootstrap complete.
I20260926 01:05:09.309113 7113 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } }
I20260926 01:05:09.309206 7113 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7cda31285d14786b026cc33437c40e7, State: Initialized, Role: LEARNER
I20260926 01:05:09.309248 7113 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } }
I20260926 01:05:09.309441 7113 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:09.309655 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } } }
I20260926 01:05:09.309759 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: This master's current role is: LEARNER
I20260926 01:05:09.310997 7127 catalog_manager.cc:1259] Loaded cluster ID: e6ab4cb480ae405ab602f8b999ba1c19
I20260926 01:05:09.311050 7127 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: loading cluster ID for follower catalog manager: success
I20260926 01:05:09.312172 7127 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: acquiring CA information for follower catalog manager: success
I20260926 01:05:09.312448 7127 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:05:09.327638 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.320417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.0.98.61:43727)
0926 01:05:09.320543 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.320545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.320552 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:05:09.320572 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.320573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.320583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.320635 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.321179 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.321830 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.322765 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.323021 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.323084 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.324399 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.324400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.324401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.324409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.326000 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.326573 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.326575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.326575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.326651 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.326954 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.326955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.326956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.327108 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.327217 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.327361 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.327516 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 01:05:09.353719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38603
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35137,127.0.98.61:43727,127.0.98.60:38603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:09.441512 7131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:09.441753 7131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:09.441778 7131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:09.443086 7131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:09.443142 7131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:09.443157 7131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:09.443171 7131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:09.443183 7131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:09.444795 7131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35137,127.0.98.61:43727,127.0.98.60:38603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:09.445061 7131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:09.445309 7131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:09.450299 7136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.450295 7137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.450299 7139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.450608 7131 server_base.cc:1034] running on GCE node
I20260926 01:05:09.450894 7131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:09.451177 7131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:09.452369 7131 hybrid_clock.cc:648] HybridClock initialized: now 1790384709452344 us; error 33 us; skew 500 ppm
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:09.455066 7131 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 01:05:09.455569 7131 webserver.cc:460] Webserver started at http://127.0.98.60:43149/ using document root <none> and password file <none>
I20260926 01:05:09.455816 7131 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:09.455873 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:09.455997 7131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:09.457114 7131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data/instance:
uuid: "1edf6b838f9d4255bb2ee0aac9a9913a"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "69ccb43e1871b57190e29d20a98c47f1"
server_key_iv: "11804fbfb35c1e800ce58659f297e08a"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.457472 7131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal/instance:
uuid: "1edf6b838f9d4255bb2ee0aac9a9913a"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "69ccb43e1871b57190e29d20a98c47f1"
server_key_iv: "11804fbfb35c1e800ce58659f297e08a"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.458925 7131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:05:09.459672 7146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.459818 7131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:05:09.459940 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal
uuid: "1edf6b838f9d4255bb2ee0aac9a9913a"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "69ccb43e1871b57190e29d20a98c47f1"
server_key_iv: "11804fbfb35c1e800ce58659f297e08a"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.460018 7131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:09.477180 7131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:09.479471 7131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:09.479588 7131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:09.483997 7131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38603
I20260926 01:05:09.484042 7198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38603 every 8 connection(s)
I20260926 01:05:09.484368 7131 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
I20260926 01:05:09.485364 7199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:09.486958 7199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:05:09.489135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 7131
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 01:05:09.497620 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.487852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.0.98.62:35137)
0926 01:05:09.487940 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.487942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.488067 (+ 125us) server_negotiation.cc:407] Connection header received
0926 01:05:09.488574 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.488575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.488585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.488643 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.489509 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.490182 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.491730 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.491995 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.492084 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.494752 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.494754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.494755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.494763 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.496204 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.496653 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.496654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.496655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.496707 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.496940 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.496941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.496942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.497083 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.497153 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.497446 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.497523 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 01:05:09.497822 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.487769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:43166)
0926 01:05:09.487941 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.487985 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.488443 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.488720 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.488724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.489071 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.489366 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.489372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.490303 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.490425 (+ 122us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.491626 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.491628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.492129 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.492130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.492251 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.492818 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.492828 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.494618 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.496271 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.496279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.496282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.496577 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.496768 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.496769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.496769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.496880 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.497168 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.497174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.497300 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.497411 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.497565 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260926 01:05:09.498356 7199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:05:09.498937 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.489688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.0.98.60:38603)
0926 01:05:09.489849 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.489854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.489868 (+ 14us) server_negotiation.cc:407] Connection header received
0926 01:05:09.489886 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.489888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.489903 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.489965 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.490683 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.491806 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.492633 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.493083 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.493220 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.495911 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.495921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.495921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.495929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.497462 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.497856 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.497860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.497860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.497920 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.498148 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.498148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.498149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.498316 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.498480 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.498791 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.498862 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 01:05:09.504907 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.498707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.0.98.61:43727)
0926 01:05:09.498747 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.498749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.498792 (+ 43us) server_negotiation.cc:407] Connection header received
0926 01:05:09.498859 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.498860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.498873 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.498918 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.499203 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.499779 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.500427 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.500635 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.500688 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.502355 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.502356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.502356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.502364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.503794 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.504146 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.504147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.504148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.504209 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.504391 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.504392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.504392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.504516 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.504579 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.504724 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.504810 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:05:09.504920 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.498650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43727 (local address 127.0.0.1:59710)
0926 01:05:09.498748 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.498759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.498805 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.498918 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.498919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.499054 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.499112 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.499113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.499843 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.499844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.500357 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.500359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.500689 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.500690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.500754 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.501139 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.501143 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.502269 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.503902 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.503903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.503903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.504078 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.504269 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.504270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.504270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.504333 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.504575 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.504576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.504626 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.504695 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.504802 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 01:05:09.505460 7199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 01:05:09.512089 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.505790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.0.98.60:38603)
0926 01:05:09.505837 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.505839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.505879 (+ 40us) server_negotiation.cc:407] Connection header received
0926 01:05:09.505935 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.505935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.505946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.505991 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.506252 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.506806 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.507426 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.507630 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.507686 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.509553 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.509554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.509555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.509564 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.510982 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.511309 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.511310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.511310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.511405 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.511615 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.511616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.511616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.511720 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.511772 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.511936 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.512020 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":27}
I20260926 01:05:09.512246 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.505736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38603 (local address 127.0.0.1:48862)
0926 01:05:09.505834 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.505844 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.505894 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.505997 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.505997 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.506153 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.506187 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.506188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.506877 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.506878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.507359 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.507360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.507687 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.507688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.507731 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.508197 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.508201 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.509456 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.511068 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.511069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.511069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.511225 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.511509 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.511510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.511510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.511567 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.511792 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.511793 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.511836 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.511919 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.512147 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":123}
I20260926 01:05:09.513734 7199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: Bootstrap starting.
I20260926 01:05:09.514591 7199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:09.514835 7199 log.cc:824] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:09.515676 7199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: No bootstrap required, opened a new log
I20260926 01:05:09.517158 7199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } }
I20260926 01:05:09.517305 7199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:09.517338 7199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edf6b838f9d4255bb2ee0aac9a9913a, State: Initialized, Role: FOLLOWER
I20260926 01:05:09.517475 7199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } }
I20260926 01:05:09.517804 7199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:09.517808 7204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } } }
I20260926 01:05:09.517930 7204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:05:09.519457 7215 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:05:09.519524 7215 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:05:09.522168 7131 master_runner.cc:428] Detected that this master 1edf6b838f9d4255bb2ee0aac9a9913a is joining an existing cluster
I20260926 01:05:09.522233 7131 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:38603
I20260926 01:05:09.549559 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.524288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.0.98.62:35137)
0926 01:05:09.524388 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.524390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.524516 (+ 126us) server_negotiation.cc:407] Connection header received
0926 01:05:09.524597 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.524598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.524608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.524667 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.524979 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.525490 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.526706 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.527010 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.527050 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.528136 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.528138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.528138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.528146 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.529534 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.547453 (+ 17919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.547456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.547456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.548038 (+ 582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.548904 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.548906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.548906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.549097 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.549149 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.549368 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.549458 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 01:05:09.550012 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.523956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:43168)
0926 01:05:09.524442 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.524453 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.524528 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.524678 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.524679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.524852 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.524905 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.524907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.525568 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.525569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.526598 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.526600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.527123 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.527124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.527188 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.527668 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.527673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.528027 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.529634 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.529636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.529636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.547159 (+ 17523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.548181 (+ 1022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.548182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.548182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.548801 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.549202 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.549203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.549252 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.549805 (+ 553us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.549912 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":29,"thread_start_us":77,"threads_started":1}
I20260926 01:05:09.550282 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.524393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.0.98.61:43727)
0926 01:05:09.524440 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.524442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.526553 (+ 2111us) server_negotiation.cc:407] Connection header received
0926 01:05:09.526647 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.526648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.526660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.526689 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.527118 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.527869 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.528932 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.529229 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.529272 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.547273 (+ 18001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.547275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.547276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.547284 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.548938 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.549424 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.549425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.549425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.549482 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.549733 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.549734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.549735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.549864 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.549906 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.550101 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.550184 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:05:09.550282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 6542
I20260926 01:05:09.550731 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.524155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43727 (local address 127.0.0.1:59712)
0926 01:05:09.526488 (+ 2333us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.526498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.526579 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.526751 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.526751 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.526970 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.527028 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.527029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.527968 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.527969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.528829 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.528832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.529362 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.529362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.529424 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.529810 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.529815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.547126 (+ 17311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.549035 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.549036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.549037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.549330 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.549550 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.549550 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.549551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.549605 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.549953 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.549954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.549998 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.550525 (+ 527us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.550633 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2314,"thread_start_us":74,"threads_started":1}
I20260926 01:05:09.555660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43003
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35137
--webserver_port=37287
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35137,127.0.98.61:43727
--tserver_master_addrs=127.0.98.62:35137,127.0.98.61:43727,127.0.98.60:38603 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:05:09.556344 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.524106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.0.98.60:38603)
0926 01:05:09.524141 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.524143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.524152 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:09.524233 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.524234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.524243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.524302 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.524821 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.525580 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.526704 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.526981 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.527024 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.547491 (+ 20467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.547492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.547492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.547498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.549504 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.555535 (+ 6031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.555536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.555536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.555627 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.555806 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.555807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.555807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.555938 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.555989 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.556157 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.556246 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:05:09.556768 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.523930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38603 (local address 127.0.0.1:48866)
0926 01:05:09.524094 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.524105 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.524158 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.524328 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.524329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.524646 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.524741 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.524743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.525655 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.525656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.526598 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.526600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.527115 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.527116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.527174 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.546983 (+ 19809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.546991 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.547405 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.549588 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.549588 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.549589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.555432 (+ 5843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.555693 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.555693 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.555694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.555749 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.556015 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.556016 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.556052 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.556457 (+ 405us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.556642 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":38,"thread_start_us":90,"threads_started":1}
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 01:05:09.565232 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.555536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38603 (local address 127.0.0.1:48878)
0926 01:05:09.555656 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.555667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.555706 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.556587 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.556588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.556767 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.556837 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.556840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.557918 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.557920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.558791 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.558794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.559309 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.559310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.559387 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.559981 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.559992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.561771 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.564054 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.564055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.564056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.564295 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.564522 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.564523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.564524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.564589 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.564875 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.564876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.564934 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.565004 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.565120 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 01:05:09.565259 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.555797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.0.98.60:38603)
0926 01:05:09.556246 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.556248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.556257 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:09.556299 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.556300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.556311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.557035 (+ 724us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.557092 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.557825 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.558878 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.559205 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.559663 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.561884 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.561886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.561886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.561895 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.563930 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.564384 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.564385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.564385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.564453 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.564650 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.564651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.564651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.564799 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.564871 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.565058 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.565147 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":76,"threads_started":1}
I20260926 01:05:09.566149 6480 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:38603: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 38603
}
attrs {
promote: true
}
}
dest_uuid: "b0573a39c6e042d19458b0849f990d9b"
cas_config_opid_index: 5
I20260926 01:05:09.566563 6480 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } }
I20260926 01:05:09.567118 7038 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.568133 7038 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:38603
I20260926 01:05:09.568210 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.568271 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:09.568452 7131 master.cc:537] Master@127.0.98.60:38603 shutting down...
W20260926 01:05:09.570004 6466 consensus_peers.cc:487] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b -> Peer 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60:38603): Couldn't send request to peer 1edf6b838f9d4255bb2ee0aac9a9913a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:05:09.572480 7131 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:09.572597 7131 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:09.572642 7131 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: stopping tablet replica
I20260926 01:05:09.574769 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.568779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.0.98.61:43727)
0926 01:05:09.569466 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.569468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.569553 (+ 85us) server_negotiation.cc:407] Connection header received
0926 01:05:09.569655 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.569656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.569678 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.569717 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:09.569914 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.570836 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.573347 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.574124 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.574184 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.574255 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.574646 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.574693 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662}
I20260926 01:05:09.575138 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.568581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43727 (local address 127.0.0.1:59716)
0926 01:05:09.569498 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.569509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.569583 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.569799 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.569800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.569802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:09.569848 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.569850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.571226 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.571227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.573216 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.573225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.574326 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.574327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.574527 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.574557 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.574994 (+ 437us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.575066 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898}
I20260926 01:05:09.575388 7087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 LEARNER]: Refusing update from remote peer b0573a39c6e042d19458b0849f990d9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 01:05:09.575742 7039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 01:05:09.576392 7087 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.577559 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0573a39c6e042d19458b0849f990d9b. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.577648 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: This master's current role is: LEARNER
I20260926 01:05:09.577678 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: true } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.577719 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: This master's current role is: LEARNER
I20260926 01:05:09.578003 7038 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b: attempting to promote NON_VOTER d7cda31285d14786b026cc33437c40e7 to VOTER
I20260926 01:05:09.578227 7038 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } }
I20260926 01:05:09.578656 7087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 LEARNER]: Refusing update from remote peer b0573a39c6e042d19458b0849f990d9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 01:05:09.578840 7039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 01:05:09.578912 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.578685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38603 (local address 127.0.0.1:48890)
0926 01:05:09.578795 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.578837 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 01:05:09.579432 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.579317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38603 (local address 127.0.0.1:48906)
0926 01:05:09.579374 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.579406 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 01:05:09.579716 6466 consensus_peers.cc:487] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b -> Peer 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60:38603): Couldn't send request to peer 1edf6b838f9d4255bb2ee0aac9a9913a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:05:09.579696 7039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d7cda31285d14786b026cc33437c40e7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.580571 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.580646 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:09.580689 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.580730 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:09.580967 7087 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d7cda31285d14786b026cc33437c40e7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.582043 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.582144 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:05:09.587007 7131 master.cc:552] Master@127.0.98.60:38603 shutdown complete.
I20260926 01:05:09.587075 7131 master_runner.cc:305] Clearing existing system tablet
I20260926 01:05:09.587213 7131 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:09.587287 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:09.588145 7131 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:09.588683 7230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.588788 7131 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:09.588835 7131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:09.588865 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal
uuid: "1edf6b838f9d4255bb2ee0aac9a9913a"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "69ccb43e1871b57190e29d20a98c47f1"
server_key_iv: "11804fbfb35c1e800ce58659f297e08a"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.589620 7131 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:09.591326 7131 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 01:05:09.591404 7131 log.cc:1196] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 01:05:09.591585 7131 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: Deleting consensus metadata
I20260926 01:05:09.591820 7131 master_runner.cc:315] Copying system tablet from 127.0.98.62:35137
I20260926 01:05:09.592433 7131 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35137
I20260926 01:05:09.601871 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.594732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:43178)
0926 01:05:09.594902 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.594913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.594991 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.595152 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.595153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.595355 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.595410 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.595412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.596374 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.596375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.597263 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.597266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.597749 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.597750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.597810 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.598339 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.598344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.598829 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.600581 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.600583 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.600584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.600802 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.601089 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.601090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.601091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.601157 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.601507 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.601508 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.601561 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.601637 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.601752 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260926 01:05:09.601908 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.594792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.0.98.62:35137)
0926 01:05:09.594845 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.594847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.594996 (+ 149us) server_negotiation.cc:407] Connection header received
0926 01:05:09.595057 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.595058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.595070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.595139 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.595498 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.596280 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.597361 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.597667 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.597741 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.598926 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.598928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.598928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.598938 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.600491 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.600889 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.600890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.600890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.600957 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.601223 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.601224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.601224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.601382 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.601457 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.601677 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.601764 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:05:09.602665 6498 tablet_copy_service.cc:140] P b0573a39c6e042d19458b0849f990d9b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1edf6b838f9d4255bb2ee0aac9a9913a ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43178)
I20260926 01:05:09.602737 6498 tablet_copy_service.cc:161] P b0573a39c6e042d19458b0849f990d9b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1edf6b838f9d4255bb2ee0aac9a9913a at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43178: session id = 1edf6b838f9d4255bb2ee0aac9a9913a-00000000000000000000000000000000
I20260926 01:05:09.603319 6498 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 01:05:09.603826 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 01:05:09.605192 7131 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: tablet copy: Starting download of 0 data blocks...
I20260926 01:05:09.605348 7131 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: tablet copy: Starting download of 1 WAL segments...
I20260926 01:05:09.606350 7131 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 01:05:09.607594 6498 tablet_copy_service.cc:342] P b0573a39c6e042d19458b0849f990d9b: Request end of tablet copy session 1edf6b838f9d4255bb2ee0aac9a9913a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43178
I20260926 01:05:09.607666 6498 tablet_copy_service.cc:433] P b0573a39c6e042d19458b0849f990d9b: ending tablet copy session 1edf6b838f9d4255bb2ee0aac9a9913a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1edf6b838f9d4255bb2ee0aac9a9913a
I20260926 01:05:09.613304 7131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:09.615422 7237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.616026 7238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.616070 7131 server_base.cc:1034] running on GCE node
W20260926 01:05:09.616535 7240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.616803 7131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:09.616982 7131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:09.618126 7131 hybrid_clock.cc:648] HybridClock initialized: now 1790384709618109 us; error 36 us; skew 500 ppm
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:09.619792 7131 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 01:05:09.620091 7131 webserver.cc:460] Webserver started at http://127.0.98.60:45783/ using document root <none> and password file <none>
I20260926 01:05:09.620208 7131 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:09.620244 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:09.621045 7131 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:09.621658 7247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.621841 7131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.621915 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal
uuid: "1edf6b838f9d4255bb2ee0aac9a9913a"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "69ccb43e1871b57190e29d20a98c47f1"
server_key_iv: "11804fbfb35c1e800ce58659f297e08a"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.621981 7131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:09.636433 7131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:09.638345 7131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:09.641556 7131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38603
I20260926 01:05:09.641652 7299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38603 every 8 connection(s)
I20260926 01:05:09.641913 7131 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
I20260926 01:05:09.643011 7300 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:05:09.643644 7300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: Bootstrap starting.
I20260926 01:05:09.647298 7300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:05:09.647552 7300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: Bootstrap complete.
I20260926 01:05:09.647823 7300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } }
I20260926 01:05:09.647950 7300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1edf6b838f9d4255bb2ee0aac9a9913a, State: Initialized, Role: LEARNER
I20260926 01:05:09.648002 7300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } }
I20260926 01:05:09.648217 7300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:09.648520 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:09.648751 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: This master's current role is: LEARNER
I20260926 01:05:09.650604 7314 catalog_manager.cc:1259] Loaded cluster ID: e6ab4cb480ae405ab602f8b999ba1c19
I20260926 01:05:09.650669 7314 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: loading cluster ID for follower catalog manager: success
I20260926 01:05:09.651813 7314 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: acquiring CA information for follower catalog manager: success
I20260926 01:05:09.652160 7314 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 01:05:09.664960 7225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:09.665208 7225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:09.665237 7225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:09.666566 7225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:09.666622 7225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:09.666657 7225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:09.668411 7225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37287
--tserver_master_addrs=127.0.98.62:35137,127.0.98.61:43727,127.0.98.60:38603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:09.668666 7225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:09.668931 7225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:09.675386 7319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.675392 7320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.676122 7225 server_base.cc:1034] running on GCE node
W20260926 01:05:09.678534 7322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.678829 7225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:09.679111 7225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:09.680275 7225 hybrid_clock.cc:648] HybridClock initialized: now 1790384709680259 us; error 36 us; skew 500 ppm
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:09.682739 7225 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 01:05:09.683207 7225 webserver.cc:460] Webserver started at http://127.0.98.1:37287/ using document root <none> and password file <none>
I20260926 01:05:09.683413 7225 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:09.683471 7225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:09.684897 7225 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:09.685530 7329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.685698 7225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:05:09.685791 7225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "478ce1ac7e994e7db5035b881a7bfa00"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "287d0f58cf43b0d28abfc4ac8f453eb9"
server_key_iv: "464c72a58a4175f7e4c3d08252b5a655"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.686144 7225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:09.722301 7225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:09.725312 7225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:09.725492 7225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:09.725741 7225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:09.726066 7225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:09.726106 7225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.726141 7225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:09.726166 7225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.736186 7225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43003
I20260926 01:05:09.736256 7442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43003 every 8 connection(s)
I20260926 01:05:09.736826 7225 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:05:09.744810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 7225
I20260926 01:05:09.744982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 6676
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 01:05:09.750734 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.738013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59741 (local address 127.0.98.62:35137)
0926 01:05:09.738079 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.738081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.738852 (+ 771us) server_negotiation.cc:407] Connection header received
0926 01:05:09.739811 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.739812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.739827 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.739855 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.740924 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.741707 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.742958 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.743238 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.743304 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.747135 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.747136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.747137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.747144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.749151 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.749600 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.749601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.749601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.749687 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.749969 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.749970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.749970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.750122 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.750177 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.750543 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.750634 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:05:09.750780 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.738503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39235 (local address 127.0.98.60:38603)
0926 01:05:09.738696 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.738698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.738707 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:09.739814 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.739816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.739831 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.739900 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.741261 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.742142 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.743118 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.743349 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.743417 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.747168 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.747170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.747171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.747179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.748890 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.749474 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.749476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.749477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.749550 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.749856 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.749857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.749858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.750073 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.750131 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.750562 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.750670 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":61,"threads_started":1}
I20260926 01:05:09.750777 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.738755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41045 (local address 127.0.98.61:43727)
0926 01:05:09.738800 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.738802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.738923 (+ 121us) server_negotiation.cc:407] Connection header received
0926 01:05:09.739817 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.739818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.739831 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.739863 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.740924 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.741707 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.743077 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.743365 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.743421 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.747013 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.747016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.747016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.747026 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.748650 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.749695 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.749697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.749698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.749760 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.750099 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.750100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.750100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.750256 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.750303 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.750608 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.750688 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:05:09.751416 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.737988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:59741)
0926 01:05:09.738785 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.738797 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.739680 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.740001 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.740004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.740524 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.740768 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.740778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.741832 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.741833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.742875 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.742878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.743317 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.743318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.743470 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.744144 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.744155 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.747034 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.749316 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.749318 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.749318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.749528 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.749820 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.749820 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.749821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.749878 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.750206 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.750207 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.750359 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.750871 (+ 512us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.751110 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":63,"spinlock_wait_cycles":384,"thread_start_us":219,"threads_started":1}
I20260926 01:05:09.751420 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.737988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38603 (local address 127.0.98.1:39235)
0926 01:05:09.738319 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.738371 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.739676 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.739999 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.740004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.740523 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.741181 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.741183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.742233 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.742234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.743060 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.743062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.743424 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.743425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.743491 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.744254 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.744259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.747051 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.748972 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.748973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.748974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.749356 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.749626 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.749627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.749627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.749776 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.750168 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.750175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.750360 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.750885 (+ 525us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.751196 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":144,"thread_start_us":219,"threads_started":1}
I20260926 01:05:09.751945 7444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35137
I20260926 01:05:09.752080 7444 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:09.752090 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.738657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43727 (local address 127.0.98.1:41045)
0926 01:05:09.738842 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.738851 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.739705 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.740004 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.740005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.740539 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.740767 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.740773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.741834 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.741835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.742991 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.742993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.743469 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.743470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.743522 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.744267 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.744271 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.746869 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.748760 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.748773 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.748776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.749611 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.749959 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.749959 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.749960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.750021 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.750352 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.750353 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.750488 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.751404 (+ 916us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.751940 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":35,"thread_start_us":84,"threads_started":1}
I20260926 01:05:09.752276 7444 heartbeater.cc:511] Master 127.0.98.62:35137 requested a full tablet report, sending...
I20260926 01:05:09.752815 6480 ts_manager.cc:194] Re-registered known tserver with Master: 478ce1ac7e994e7db5035b881a7bfa00 (127.0.98.1:43003)
I20260926 01:05:09.753404 6480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59741
I20260926 01:05:09.753472 7445 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43727
I20260926 01:05:09.753533 7445 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:09.753553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39833
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35137
--webserver_port=35757
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35137,127.0.98.61:43727
--tserver_master_addrs=127.0.98.62:35137,127.0.98.61:43727,127.0.98.60:38603 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:05:09.753536 7443 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38603
I20260926 01:05:09.753615 7445 heartbeater.cc:511] Master 127.0.98.61:43727 requested a full tablet report, sending...
I20260926 01:05:09.753664 7443 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:09.753732 7443 heartbeater.cc:511] Master 127.0.98.60:38603 requested a full tablet report, sending...
I20260926 01:05:09.754140 7264 ts_manager.cc:194] Registered new tserver with Master: 478ce1ac7e994e7db5035b881a7bfa00 (127.0.98.1:43003)
I20260926 01:05:09.754333 7077 ts_manager.cc:194] Registered new tserver with Master: 478ce1ac7e994e7db5035b881a7bfa00 (127.0.98.1:43003)
W20260926 01:05:09.840284 7451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:09.840531 7451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:09.840559 7451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:09.841823 7451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:09.841886 7451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:09.841920 7451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:09.843597 7451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35757
--tserver_master_addrs=127.0.98.62:35137,127.0.98.61:43727,127.0.98.60:38603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:09.843853 7451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:09.844107 7451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:09.849663 7456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.849658 7457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.849663 7459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.850273 7451 server_base.cc:1034] running on GCE node
I20260926 01:05:09.850566 7451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:09.850812 7451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:09.852013 7451 hybrid_clock.cc:648] HybridClock initialized: now 1790384709851995 us; error 34 us; skew 500 ppm
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:09.855029 7451 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 01:05:09.855579 7451 webserver.cc:460] Webserver started at http://127.0.98.2:35757/ using document root <none> and password file <none>
I20260926 01:05:09.855810 7451 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:09.855875 7451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:09.857537 7451 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:09.858342 7466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.858624 7451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:09.858722 7451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "32e08a68241a46fb9685683facdbaf77"
format_stamp: "Formatted at 2026-09-26 01:05:08 on dist-test-slave-l73v"
server_key: "0b9c88e05de18d6b2b343911070a2385"
server_key_iv: "121278c8af925e131cd74cec840ab8e8"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.859151 7451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:09.872068 7451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:09.874428 7451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:09.874571 7451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:09.874809 7451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:09.875123 7451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:09.875157 7451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.875187 7451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:09.875211 7451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.881642 7451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39833
I20260926 01:05:09.881690 7579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39833 every 8 connection(s)
I20260926 01:05:09.882064 7451 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:05:09.888933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 7451
I20260926 01:05:09.889103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 6810
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 01:05:09.894624 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.883376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42065 (local address 127.0.98.62:35137)
0926 01:05:09.883466 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.883467 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.883475 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:09.884449 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.884451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.884466 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.884584 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.886533 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.887090 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.888105 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.888365 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.888435 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.891068 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.891070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.891071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.891079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.892817 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.893504 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.893505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.893506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.893565 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.893857 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.893858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.893859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.893996 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.894040 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.894439 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.894528 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:05:09.894760 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.882863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52595 (local address 127.0.98.61:43727)
0926 01:05:09.882926 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.882928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.883175 (+ 247us) server_negotiation.cc:407] Connection header received
0926 01:05:09.884272 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.884273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.884289 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.884347 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.885977 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.886782 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.887657 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.887985 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.888057 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.891573 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.891575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.891575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.891583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.893353 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.893733 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.893735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.893735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.893791 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.894069 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.894070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.894071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.894210 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.894272 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.894578 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.894669 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:05:09.895102 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.883188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:42065)
0926 01:05:09.883317 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.883326 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.884157 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.884598 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.884599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.884967 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.885267 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.885273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.887167 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.887168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.887984 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.887986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.888487 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.888488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.888563 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.889113 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.889117 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.890951 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.893057 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.893070 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.893073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.893412 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.893629 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.893630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.893630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.893774 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.894087 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.894094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.894275 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.894407 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.894584 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":29,"thread_start_us":68,"threads_started":1}
I20260926 01:05:09.895190 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.882879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43727 (local address 127.0.98.2:52595)
0926 01:05:09.883049 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.883091 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.884144 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.884513 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.884517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.884961 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.885267 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.885273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.886874 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.886875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.887575 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.887577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.888293 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.888294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.888366 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.888873 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.888883 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.891480 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.893446 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.893447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.893448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.893651 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.893849 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.893850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.893850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.893911 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.894273 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.894274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.894326 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.894846 (+ 520us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.894954 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":80,"thread_start_us":90,"threads_started":1}
I20260926 01:05:09.895561 7581 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35137
I20260926 01:05:09.895681 7581 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:09.895746 7582 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43727
I20260926 01:05:09.895750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40797
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35137
--webserver_port=46705
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35137,127.0.98.61:43727
--tserver_master_addrs=127.0.98.62:35137,127.0.98.61:43727,127.0.98.60:38603 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:05:09.895799 7582 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:09.895905 7582 heartbeater.cc:511] Master 127.0.98.61:43727 requested a full tablet report, sending...
I20260926 01:05:09.895903 7581 heartbeater.cc:511] Master 127.0.98.62:35137 requested a full tablet report, sending...
I20260926 01:05:09.896324 6480 ts_manager.cc:194] Re-registered known tserver with Master: 32e08a68241a46fb9685683facdbaf77 (127.0.98.2:39833)
I20260926 01:05:09.896380 7077 ts_manager.cc:194] Registered new tserver with Master: 32e08a68241a46fb9685683facdbaf77 (127.0.98.2:39833)
I20260926 01:05:09.896788 6480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42065
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 01:05:09.901525 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.882879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38603 (local address 127.0.98.2:52191)
0926 01:05:09.883048 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:09.883091 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:09.884164 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:09.884514 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:09.884519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:09.884953 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:09.885266 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.885272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.885986 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.885987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.887409 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:09.887411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.887844 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.887845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:09.888030 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.896988 (+ 8958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:09.897052 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:09.898561 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:09.900504 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.900506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.900506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.900687 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.900898 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:09.900898 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:09.900899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:09.900948 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:09.901219 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:09.901220 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:09.901261 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 01:05:09.901312 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 01:05:09.901427 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260926 01:05:09.901535 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:09.883131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52191 (local address 127.0.98.60:38603)
0926 01:05:09.883172 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:09.883174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:09.883183 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:09.884348 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:09.884349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:09.884364 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:09.884408 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:09.885362 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.885877 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.887518 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:09.887772 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:09.887830 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:09.898662 (+ 10832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:09.898664 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:09.898665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:09.898673 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:09.900387 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.900764 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.900765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.900766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.900831 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:09.901004 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:09.901005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:09.901005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:09.901155 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:09.901232 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:09.901337 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 01:05:09.901428 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:05:09.902439 7580 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38603
I20260926 01:05:09.902514 7580 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:09.902599 7580 heartbeater.cc:511] Master 127.0.98.60:38603 requested a full tablet report, sending...
I20260926 01:05:09.902945 7264 ts_manager.cc:194] Registered new tserver with Master: 32e08a68241a46fb9685683facdbaf77 (127.0.98.2:39833)
W20260926 01:05:09.979791 7587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:09.980032 7587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:09.980068 7587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:09.981326 7587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:09.981376 7587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:09.981415 7587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:09.983026 7587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46705
--tserver_master_addrs=127.0.98.62:35137,127.0.98.61:43727,127.0.98.60:38603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:09.983280 7587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:09.983506 7587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:09.988941 7593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:09.988978 7595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.989046 7587 server_base.cc:1034] running on GCE node
W20260926 01:05:09.988984 7592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:09.989332 7587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:09.989570 7587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:09.990722 7587 hybrid_clock.cc:648] HybridClock initialized: now 1790384709990695 us; error 43 us; skew 500 ppm
Sep 26 01:05:09 dist-test-slave-l73v krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:09.993321 7587 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 01:05:09.993794 7587 webserver.cc:460] Webserver started at http://127.0.98.3:46705/ using document root <none> and password file <none>
I20260926 01:05:09.994015 7587 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:09.994072 7587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:09.995683 7587 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:09.996405 7602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:09.996647 7587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:09.996734 7587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "edc5dc1a23274a64aa4b90e7f583dbeb"
format_stamp: "Formatted at 2026-09-26 01:05:09 on dist-test-slave-l73v"
server_key: "2f9e497f743ecdd6d3610f608f6159e9"
server_key_iv: "4c2bf730e6ca025be015f3722bbb41a6"
server_key_version: "encryptionkey@0"
I20260926 01:05:09.997143 7587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:10.019702 7587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:10.022311 7587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:10.022535 7587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:10.022816 7587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:10.023147 7587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:10.023185 7587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:10.023219 7587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:10.023243 7587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:10.029471 7587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40797
I20260926 01:05:10.029527 7715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40797 every 8 connection(s)
I20260926 01:05:10.029893 7587 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:05:10.031731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 7587
Sep 26 01:05:10 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 01:05:10 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 01:05:10 dist-test-slave-l73v krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 01:05:10.041208 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.030769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34171 (local address 127.0.98.61:43727)
0926 01:05:10.030838 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.030840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.030994 (+ 154us) server_negotiation.cc:407] Connection header received
0926 01:05:10.031566 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.031567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.031580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.031621 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:10.032861 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.033582 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.035154 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.035395 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.035452 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.038135 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:10.038136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:10.038137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:10.038143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:10.039670 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.040142 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.040143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.040143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.040212 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.040510 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.040511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.040512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.040638 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:10.040680 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.041040 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.041111 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:05:10.041491 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.032651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.0.98.60:38603)
0926 01:05:10.032878 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.032880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.033021 (+ 141us) server_negotiation.cc:407] Connection header received
0926 01:05:10.033040 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.033041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.033052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.033100 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:10.033831 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.034526 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.035236 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.035538 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.035606 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.037948 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:10.037950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:10.037950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:10.037958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:10.039990 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.040358 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.040359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.040360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.040424 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.040628 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.040629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.040629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.040758 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:10.040849 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.041305 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.041389 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":48,"threads_started":1}
I20260926 01:05:10.041749 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.030679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43727 (local address 127.0.98.3:34171)
0926 01:05:10.030863 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:10.030906 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:10.031479 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:10.032527 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:10.032528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:10.032697 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:10.032766 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.032768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.033679 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.033679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.035061 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.035064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.035478 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.035479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.035628 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.036185 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:10.036196 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:10.038048 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:10.039757 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:10.039764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:10.039766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:10.040055 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:10.040282 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:10.040284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:10.040284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:10.040416 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:10.040726 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:10.040732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:10.040895 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 01:05:10.041331 (+ 436us) client_negotiation.cc:240] Negotiation successful
0926 01:05:10.041522 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":90,"spinlock_wait_cycles":7552,"thread_start_us":104,"threads_started":1}
I20260926 01:05:10.042153 7718 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43727
I20260926 01:05:10.042243 7718 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:10.042471 7718 heartbeater.cc:511] Master 127.0.98.61:43727 requested a full tablet report, sending...
I20260926 01:05:10.042879 7077 ts_manager.cc:194] Registered new tserver with Master: edc5dc1a23274a64aa4b90e7f583dbeb (127.0.98.3:40797)
I20260926 01:05:10.044268 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.030970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:37763)
0926 01:05:10.031104 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:10.031114 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:10.031485 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:10.032175 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:10.032176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:10.032364 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:10.032599 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.032605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.033639 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.033640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.035826 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.035829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.036299 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.036300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.036360 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.036735 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:10.036740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:10.039086 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:10.040958 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:10.040959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:10.040959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:10.041143 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:10.041701 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:10.041702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:10.041702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:10.043670 (+ 1968us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:10.043962 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:10.043963 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:10.044004 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 01:05:10.044065 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 01:05:10.044169 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":87,"thread_start_us":78,"threads_started":1}
I20260926 01:05:10.044625 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.030933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32981 (local address 127.0.98.60:38603)
0926 01:05:10.030980 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.030981 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.030992 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:10.031602 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.031603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.031617 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.031651 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:10.032720 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.033487 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.035507 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.035785 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.036287 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.041546 (+ 5259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:10.041547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:10.041548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:10.041554 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:10.043526 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.043868 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.043869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.043869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.043917 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.044096 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.044097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.044097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.044216 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:10.044255 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.044472 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.044543 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:05:10.044754 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.031145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37763 (local address 127.0.98.62:35137)
0926 01:05:10.031187 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.031189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.031197 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:10.031602 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.031603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.031617 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.031651 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:10.032720 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.033496 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.035906 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.036187 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.036246 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.039353 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:10.039354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:10.039355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:10.039362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:10.040873 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.041212 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.041213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.041213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.041276 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.043742 (+ 2466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.043743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.043743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.043871 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:10.043926 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.044551 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.044660 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:05:10.044936 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.030680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38603 (local address 127.0.98.3:32981)
0926 01:05:10.030895 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:10.030915 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:10.031471 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:10.031838 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:10.031842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:10.032284 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:10.032599 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.032605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.033639 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.033640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.035061 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.035064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.036331 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.036331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.036382 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.036849 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:10.036853 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:10.040169 (+ 3316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:10.043600 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:10.043601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:10.043601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:10.043794 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:10.043981 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:10.043981 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:10.043982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:10.044032 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:10.044305 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:10.044306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:10.044357 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 01:05:10.044723 (+ 366us) client_negotiation.cc:240] Negotiation successful
0926 01:05:10.044827 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":240,"thread_start_us":93,"threads_started":1}
I20260926 01:05:10.045393 7717 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35137
I20260926 01:05:10.045442 7717 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:10.045506 7717 heartbeater.cc:511] Master 127.0.98.62:35137 requested a full tablet report, sending...
I20260926 01:05:10.045727 7716 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38603
I20260926 01:05:10.045766 6480 ts_manager.cc:194] Re-registered known tserver with Master: edc5dc1a23274a64aa4b90e7f583dbeb (127.0.98.3:40797)
I20260926 01:05:10.045794 7716 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:10.045867 7716 heartbeater.cc:511] Master 127.0.98.60:38603 requested a full tablet report, sending...
I20260926 01:05:10.046186 6480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37763
I20260926 01:05:10.046375 7264 ts_manager.cc:194] Registered new tserver with Master: edc5dc1a23274a64aa4b90e7f583dbeb (127.0.98.3:40797)
I20260926 01:05:10.047314 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.034062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.0.98.62:35137)
0926 01:05:10.035971 (+ 1909us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.035973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.035981 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:10.036002 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.036003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.036015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.036083 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:10.036851 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.039854 (+ 3003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.041116 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.041385 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.041462 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.043068 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:10.043069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:10.043070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:10.043076 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:10.045586 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.046312 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.046313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.046313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.046401 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.046621 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.046622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.046622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.046739 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:10.046829 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.047188 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.047254 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1878,"thread_start_us":83,"threads_started":1}
I20260926 01:05:10.049362 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.034320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.0.98.61:43727)
0926 01:05:10.034545 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.034546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.042051 (+ 7505us) server_negotiation.cc:407] Connection header received
0926 01:05:10.042076 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.042076 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.042090 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.042150 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:10.042782 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.043454 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.044461 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.044741 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.044808 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.046339 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:10.046340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:10.046341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:10.046347 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:10.048160 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.048596 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.048597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.048598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.048653 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.048828 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.048829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.048829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.048948 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:10.049038 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.049180 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.049264 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":50,"threads_started":1}
I20260926 01:05:10.048801 6480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:05:10.050630 6480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:05:10.058163 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.054030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43003 (local address 127.0.0.1:35438)
0926 01:05:10.054084 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:10.054095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:10.054215 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:10.055044 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:10.055044 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:10.055046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:10.055071 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.055072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.056440 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.056441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.057177 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.057179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.057944 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.057944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.057995 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.057999 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:10.058048 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:05:10.058111 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 01:05:10.058195 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.054284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.0.98.1:43003)
0926 01:05:10.054847 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.054850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.054862 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:10.054899 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.054901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.054913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.055034 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:10.055150 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.056360 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.057257 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.057838 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.057910 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.057982 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.058062 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.058099 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":85,"threads_started":1}
I20260926 01:05:10.058710 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.054031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39833 (local address 127.0.0.1:44380)
0926 01:05:10.054324 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:10.054333 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:10.054369 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:10.054836 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:10.054837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:10.054839 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:10.054889 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.054890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.056260 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.056261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.057464 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.057466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.058471 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.058472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.058540 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.058544 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:10.058594 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:05:10.058633 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":26,"thread_start_us":153,"threads_started":1}
I20260926 01:05:10.058990 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.054477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.0.98.2:39833)
0926 01:05:10.054699 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.054704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.054714 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:10.054749 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.054751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.054762 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.054831 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:10.054940 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.056174 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.057552 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.058225 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.058303 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.058370 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.058873 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.058913 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":148,"threads_started":1}
I20260926 01:05:10.059186 7377 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc838ce09314daf81321a43fb3265d6 (DEFAULT_TABLE table=test-table [id=27f0588a9b8f4aacb478b233b03b5e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:10.059543 7377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc838ce09314daf81321a43fb3265d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:10.059569 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.054052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40797 (local address 127.0.0.1:46062)
0926 01:05:10.055107 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:10.055115 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:10.055148 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:10.055385 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:10.055386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:10.055388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:10.055432 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.055469 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.056355 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.056356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.057462 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.057465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.059309 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.059310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.059379 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.059384 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:05:10.059452 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 01:05:10.059492 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260926 01:05:10.059752 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.054344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.0.98.3:40797)
0926 01:05:10.055238 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.055241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.055248 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:05:10.055275 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.055277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.055288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.055356 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:10.055533 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.056269 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.058447 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.059232 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.059291 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.059339 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.059469 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.059505 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":76,"threads_started":1}
I20260926 01:05:10.060418 7650 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc838ce09314daf81321a43fb3265d6 (DEFAULT_TABLE table=test-table [id=27f0588a9b8f4aacb478b233b03b5e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:10.060737 7650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc838ce09314daf81321a43fb3265d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:10.063102 7742 tablet_bootstrap.cc:492] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: Bootstrap starting.
I20260926 01:05:10.063009 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc838ce09314daf81321a43fb3265d6 (DEFAULT_TABLE table=test-table [id=27f0588a9b8f4aacb478b233b03b5e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:10.063333 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc838ce09314daf81321a43fb3265d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:10.063840 7742 tablet_bootstrap.cc:654] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:10.064085 7742 log.cc:824] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:10.064709 7742 tablet_bootstrap.cc:492] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: No bootstrap required, opened a new log
I20260926 01:05:10.064770 7742 ts_tablet_manager.cc:1397] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:10.065744 7745 tablet_bootstrap.cc:492] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: Bootstrap starting.
I20260926 01:05:10.065980 7743 tablet_bootstrap.cc:492] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: Bootstrap starting.
I20260926 01:05:10.066370 7745 tablet_bootstrap.cc:654] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:10.066586 7743 tablet_bootstrap.cc:654] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:10.066514 7742 raft_consensus.cc:348] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } }
I20260926 01:05:10.066658 7742 raft_consensus.cc:374] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:10.066699 7742 raft_consensus.cc:729] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478ce1ac7e994e7db5035b881a7bfa00, State: Initialized, Role: FOLLOWER
I20260926 01:05:10.066711 7745 log.cc:824] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:10.066834 7742 consensus_queue.cc:260] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } }
I20260926 01:05:10.066862 7743 log.cc:824] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:10.067118 7742 ts_tablet_manager.cc:1428] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:05:10.067263 7444 heartbeater.cc:503] Master 127.0.98.62:35137 was elected leader, sending a full tablet report...
I20260926 01:05:10.067795 7743 tablet_bootstrap.cc:492] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: No bootstrap required, opened a new log
I20260926 01:05:10.067857 7743 ts_tablet_manager.cc:1397] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:10.067854 7745 tablet_bootstrap.cc:492] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: No bootstrap required, opened a new log
I20260926 01:05:10.067916 7745 ts_tablet_manager.cc:1397] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:05:10.070055 7743 raft_consensus.cc:348] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } }
I20260926 01:05:10.070230 7743 raft_consensus.cc:374] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:10.070255 7743 raft_consensus.cc:729] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc5dc1a23274a64aa4b90e7f583dbeb, State: Initialized, Role: FOLLOWER
I20260926 01:05:10.070371 7743 consensus_queue.cc:260] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } }
I20260926 01:05:10.070480 7745 raft_consensus.cc:348] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } }
I20260926 01:05:10.070641 7745 raft_consensus.cc:374] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:10.070684 7745 raft_consensus.cc:729] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e08a68241a46fb9685683facdbaf77, State: Initialized, Role: FOLLOWER
I20260926 01:05:10.070770 7743 ts_tablet_manager.cc:1428] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 01:05:10.070798 7745 consensus_queue.cc:260] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } }
I20260926 01:05:10.070909 7717 heartbeater.cc:503] Master 127.0.98.62:35137 was elected leader, sending a full tablet report...
I20260926 01:05:10.071036 7745 ts_tablet_manager.cc:1428] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 01:05:10.071587 7581 heartbeater.cc:503] Master 127.0.98.62:35137 was elected leader, sending a full tablet report...
W20260926 01:05:10.132885 7583 tablet.cc:2369] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:05:10.185437 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.182860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38603 (local address 127.0.0.1:48918)
0926 01:05:10.182968 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:10.182981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:10.183041 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:10.183213 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:10.183214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:10.183215 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:10.183267 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.183269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.183900 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.183901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.184635 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.184637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.185186 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.185187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.185248 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.185252 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:10.185307 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:05:10.185358 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 01:05:10.185446 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.183006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.0.98.60:38603)
0926 01:05:10.183084 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.183086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.183095 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:10.183121 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.183122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.183144 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.183202 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:10.183324 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.183826 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.184721 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.185116 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.185177 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.185255 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.185314 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.185350 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 01:05:10.185905 7274 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260926 01:05:10.186844 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0573a39c6e042d19458b0849f990d9b. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:10.186959 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: This master's current role is: LEARNER
W20260926 01:05:10.238790 7446 tablet.cc:2369] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:05:10.281018 7719 tablet.cc:2369] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:05:10.381875 7750 raft_consensus.cc:482] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:10.382079 7750 raft_consensus.cc:504] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } }
I20260926 01:05:10.382477 7750 leader_election.cc:288] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 478ce1ac7e994e7db5035b881a7bfa00 (127.0.98.1:43003), edc5dc1a23274a64aa4b90e7f583dbeb (127.0.98.3:40797)
I20260926 01:05:10.386782 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.382739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55885 (local address 127.0.98.1:43003)
0926 01:05:10.382793 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.382795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.382806 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:10.382838 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.382838 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.382851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.382918 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:10.383117 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.383858 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.385552 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.386230 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.386323 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.386336 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.386652 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.386701 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:05:10.386782 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.382723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57951 (local address 127.0.98.3:40797)
0926 01:05:10.382793 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.382795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.382807 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:10.382892 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.382893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.382906 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.382957 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:10.383117 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.383873 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.385546 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.386230 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.386323 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.386336 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.386661 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.386715 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 01:05:10.386785 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.382666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43003 (local address 127.0.98.2:55885)
0926 01:05:10.382712 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:10.382723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:10.382767 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:10.382938 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:10.382939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:10.382940 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:10.383022 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.383023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.384029 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.384030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.385431 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.385435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.386347 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.386348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.386551 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.386556 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:05:10.386626 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:05:10.386681 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":81}
I20260926 01:05:10.386798 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.382668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40797 (local address 127.0.98.2:57951)
0926 01:05:10.382762 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:10.382770 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:10.382802 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:10.382971 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:10.382972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:10.382973 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:10.383020 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.383022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.384029 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.384030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.385431 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:10.385435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.386347 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.386348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:10.386558 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.386561 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:10.386626 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 01:05:10.386681 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":64}
I20260926 01:05:10.387200 7670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc838ce09314daf81321a43fb3265d6" candidate_uuid: "32e08a68241a46fb9685683facdbaf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" is_pre_election: true
I20260926 01:05:10.387200 7397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc838ce09314daf81321a43fb3265d6" candidate_uuid: "32e08a68241a46fb9685683facdbaf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478ce1ac7e994e7db5035b881a7bfa00" is_pre_election: true
I20260926 01:05:10.387387 7670 raft_consensus.cc:2454] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e08a68241a46fb9685683facdbaf77 in term 0.
I20260926 01:05:10.387380 7397 raft_consensus.cc:2454] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e08a68241a46fb9685683facdbaf77 in term 0.
I20260926 01:05:10.387627 7469 leader_election.cc:302] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e08a68241a46fb9685683facdbaf77, 478ce1ac7e994e7db5035b881a7bfa00; no voters:
I20260926 01:05:10.387769 7750 raft_consensus.cc:2781] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:05:10.387843 7750 raft_consensus.cc:482] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:10.387871 7750 raft_consensus.cc:3037] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:10.388701 7750 raft_consensus.cc:504] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } }
I20260926 01:05:10.388870 7750 leader_election.cc:288] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [CANDIDATE]: Term 1 election: Requested vote from peers 478ce1ac7e994e7db5035b881a7bfa00 (127.0.98.1:43003), edc5dc1a23274a64aa4b90e7f583dbeb (127.0.98.3:40797)
I20260926 01:05:10.389010 7670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc838ce09314daf81321a43fb3265d6" candidate_uuid: "32e08a68241a46fb9685683facdbaf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb"
I20260926 01:05:10.389099 7670 raft_consensus.cc:3037] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:10.389139 7397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc838ce09314daf81321a43fb3265d6" candidate_uuid: "32e08a68241a46fb9685683facdbaf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478ce1ac7e994e7db5035b881a7bfa00"
I20260926 01:05:10.389231 7397 raft_consensus.cc:3037] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:10.389986 7397 raft_consensus.cc:2454] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e08a68241a46fb9685683facdbaf77 in term 1.
I20260926 01:05:10.390110 7670 raft_consensus.cc:2454] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e08a68241a46fb9685683facdbaf77 in term 1.
I20260926 01:05:10.390260 7469 leader_election.cc:302] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e08a68241a46fb9685683facdbaf77, 478ce1ac7e994e7db5035b881a7bfa00; no voters:
I20260926 01:05:10.390405 7750 raft_consensus.cc:2781] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:10.390537 7750 raft_consensus.cc:686] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 1 LEADER]: Becoming Leader. State: Replica: 32e08a68241a46fb9685683facdbaf77, State: Running, Role: LEADER
I20260926 01:05:10.390636 7750 consensus_queue.cc:237] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } }
I20260926 01:05:10.391494 6480 catalog_manager.cc:5495] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32e08a68241a46fb9685683facdbaf77 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "32e08a68241a46fb9685683facdbaf77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } health_report { overall_health: UNKNOWN } } }
I20260926 01:05:10.392071 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: true } } }
I20260926 01:05:10.392165 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:10.393015 7226 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b: attempting to promote NON_VOTER 1edf6b838f9d4255bb2ee0aac9a9913a to VOTER
I20260926 01:05:10.393162 7226 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } }
I20260926 01:05:10.393606 7087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Refusing update from remote peer b0573a39c6e042d19458b0849f990d9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 01:05:10.393682 7273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 LEARNER]: Refusing update from remote peer b0573a39c6e042d19458b0849f990d9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 01:05:10.393796 7039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 01:05:10.393884 7038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 01:05:10.394551 7759 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:10.394708 7087 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:10.394713 7273 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:10.395565 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:10.395637 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:10.395689 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:10.395751 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:10.395782 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:10.395813 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:10.395772 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:10.395865 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:05:10.396016 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0573a39c6e042d19458b0849f990d9b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:10.396116 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:05:10.419459 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.417110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.0.98.2:39833)
0926 01:05:10.417167 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.417169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.417177 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:10.417216 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.417217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.417240 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.417309 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:10.417416 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.417917 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.418701 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.418909 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.418966 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.419143 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:10.419251 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:10.419331 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.419354 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.419376 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:05:10.421321 7670 raft_consensus.cc:1261] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb [term 1 FOLLOWER]: Refusing update from remote peer 32e08a68241a46fb9685683facdbaf77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:10.421365 7397 raft_consensus.cc:1261] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00 [term 1 FOLLOWER]: Refusing update from remote peer 32e08a68241a46fb9685683facdbaf77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:10.421665 7756 consensus_queue.cc:1035] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:10.421816 7750 consensus_queue.cc:1035] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:10.424789 7764 mvcc.cc:205] Tried to move back new op lower bound from 7333415773883109376 to 7333415773760749568. Current Snapshot: MvccSnapshot[applied={T|T < 7333415773883109376}]
I20260926 01:05:10.425350 7762 mvcc.cc:205] Tried to move back new op lower bound from 7333415773883109376 to 7333415773760749568. Current Snapshot: MvccSnapshot[applied={T|T < 7333415773883109376}]
I20260926 01:05:10.425469 7763 mvcc.cc:205] Tried to move back new op lower bound from 7333415773883109376 to 7333415773760749568. Current Snapshot: MvccSnapshot[applied={T|T < 7333415773883109376}]
I20260926 01:05:10.427875 6479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:10.427996 6479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:10.429528 6479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0573a39c6e042d19458b0849f990d9b: Sending DeleteTablet for 3 replicas of tablet 9fc838ce09314daf81321a43fb3265d6
I20260926 01:05:10.429908 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc838ce09314daf81321a43fb3265d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44380
I20260926 01:05:10.430015 7650 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc838ce09314daf81321a43fb3265d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46062
I20260926 01:05:10.430109 7377 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc838ce09314daf81321a43fb3265d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35438
I20260926 01:05:10.430272 7773 tablet_replica.cc:330] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: stopping tablet replica
I20260926 01:05:10.430410 7773 raft_consensus.cc:2229] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:10.430536 7773 raft_consensus.cc:2258] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:10.430559 7774 tablet_replica.cc:330] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: stopping tablet replica
I20260926 01:05:10.430650 7774 raft_consensus.cc:2229] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:10.430897 7774 raft_consensus.cc:2258] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:10.430998 7773 ts_tablet_manager.cc:1905] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:10.431133 7772 tablet_replica.cc:330] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: stopping tablet replica
I20260926 01:05:10.431259 7772 raft_consensus.cc:2229] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:05:10.431332 7774 ts_tablet_manager.cc:1905] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:10.431422 7772 raft_consensus.cc:2258] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:10.431875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 6448
I20260926 01:05:10.434500 7774 ts_tablet_manager.cc:1918] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:10.434582 7774 log.cc:1196] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/9fc838ce09314daf81321a43fb3265d6
I20260926 01:05:10.434846 7774 ts_tablet_manager.cc:1939] T 9fc838ce09314daf81321a43fb3265d6 P 478ce1ac7e994e7db5035b881a7bfa00: Deleting consensus metadata
I20260926 01:05:10.436221 7773 ts_tablet_manager.cc:1918] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:10.436309 7773 log.cc:1196] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/9fc838ce09314daf81321a43fb3265d6
I20260926 01:05:10.436558 7773 ts_tablet_manager.cc:1939] T 9fc838ce09314daf81321a43fb3265d6 P edc5dc1a23274a64aa4b90e7f583dbeb: Deleting consensus metadata
W20260926 01:05:10.437706 7604 connection.cc:535] server connection from 127.0.0.1:46062 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:05:10.438431 7330 connection.cc:535] server connection from 127.0.0.1:35438 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:05:10.439095 7772 ts_tablet_manager.cc:1905] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:10.440970 7772 ts_tablet_manager.cc:1918] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:10.441069 7772 log.cc:1196] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/9fc838ce09314daf81321a43fb3265d6
I20260926 01:05:10.441372 7772 ts_tablet_manager.cc:1939] T 9fc838ce09314daf81321a43fb3265d6 P 32e08a68241a46fb9685683facdbaf77: Deleting consensus metadata
I20260926 01:05:10.441382 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.431495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.0.98.60:38603)
0926 01:05:10.431559 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.431561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.432052 (+ 491us) server_negotiation.cc:407] Connection header received
0926 01:05:10.432134 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.432134 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.432145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.432170 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:10.432970 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.433730 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.435064 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.435369 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.435412 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.437316 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:10.437318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:10.437318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:10.437326 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:10.439292 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.439907 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.439909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.439909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.439972 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.440332 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.440333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.440334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.440518 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:10.440632 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.441200 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.441285 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 01:05:10.441813 7468 connection.cc:441] server connection from 127.0.0.1:44380 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44380 (request call id 1) could send its response
I20260926 01:05:10.448902 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.440590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.0.98.60:38603)
0926 01:05:10.440675 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.440677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.441224 (+ 547us) server_negotiation.cc:407] Connection header received
0926 01:05:10.441316 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.441316 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.441327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.441355 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:10.442303 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.442921 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.443932 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.444258 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.444327 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.445837 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:10.445839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:10.445839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:10.445847 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:10.447687 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.448071 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.448072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.448073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.448137 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.448332 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.448332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.448333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.448473 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:10.448573 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.448750 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.448833 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 01:05:10.449692 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:10.440317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.0.98.61:43727)
0926 01:05:10.440362 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:10.440364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:10.442451 (+ 2087us) server_negotiation.cc:407] Connection header received
0926 01:05:10.442478 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:10.442479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:10.442496 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:10.442550 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:10.443154 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.443885 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.444748 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:10.445074 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:10.445155 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:10.446505 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:10.446507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:10.446508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:10.446516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:10.448308 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.448721 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.448722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.448722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.448781 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:10.449023 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:10.449024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:10.449024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:10.449156 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:10.449224 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:10.449509 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 01:05:10.449605 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:05:11.426007 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.425713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:57545)
0926 01:05:11.425866 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.425951 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":46,"threads_started":1}
I20260926 01:05:11.426007 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.425624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:50901)
0926 01:05:11.425879 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.425953 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":118,"threads_started":1}
W20260926 01:05:11.426432 7444 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35137 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
W20260926 01:05:11.426496 7717 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35137 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
I20260926 01:05:11.427129 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.426931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:59655)
0926 01:05:11.427033 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.427092 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
W20260926 01:05:11.427346 7581 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35137 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
I20260926 01:05:11.929911 7787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0573a39c6e042d19458b0849f990d9b)
I20260926 01:05:11.929929 7786 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0573a39c6e042d19458b0849f990d9b)
I20260926 01:05:11.930040 7787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } }
I20260926 01:05:11.930155 7786 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } }
I20260926 01:05:11.930303 7787 leader_election.cc:288] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137), d7cda31285d14786b026cc33437c40e7 (127.0.98.61:43727)
I20260926 01:05:11.930450 7786 leader_election.cc:288] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137), 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60:38603)
I20260926 01:05:11.930819 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.930488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58228)
0926 01:05:11.930721 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.930774 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":55,"threads_started":1}
I20260926 01:05:11.930972 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.930547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58234)
0926 01:05:11.930788 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.930843 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":52,"threads_started":1}
W20260926 01:05:11.930977 7249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
W20260926 01:05:11.931134 7062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
I20260926 01:05:11.931370 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.931284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58236)
0926 01:05:11.931321 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.931342 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 01:05:11.931455 7249 leader_election.cc:334] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
I20260926 01:05:11.931752 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.931653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58244)
0926 01:05:11.931700 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.931729 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 01:05:11.931821 7062 leader_election.cc:334] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
I20260926 01:05:11.933529 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.930677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43727 (local address 127.0.0.1:60756)
0926 01:05:11.930777 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.930786 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:11.930838 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:11.931030 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:11.931032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:11.931035 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:11.931088 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:11.931090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:11.931964 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:11.931965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:11.932785 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:11.932788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:11.933310 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:11.933311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:11.933365 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:11.933369 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:11.933414 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:05:11.933459 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260926 01:05:11.933526 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.930702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.0.98.61:43727)
0926 01:05:11.930839 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:11.930841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:11.930853 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:11.930903 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:11.930903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:11.930914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:11.930976 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:11.931159 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:11.931897 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:11.932863 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:11.933235 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:11.933296 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:11.933309 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:11.933427 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 01:05:11.933465 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":123,"thread_start_us":46,"threads_started":1}
I20260926 01:05:11.933684 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.930726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.0.98.60:38603)
0926 01:05:11.930934 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:11.930943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:11.931100 (+ 157us) server_negotiation.cc:407] Connection header received
0926 01:05:11.931125 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:11.931125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:11.931138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:11.931210 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:11.931387 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:11.932037 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:11.933002 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:11.933383 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:11.933441 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:11.933452 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:11.933571 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 01:05:11.933611 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":71,"threads_started":1}
I20260926 01:05:11.933923 7087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d7cda31285d14786b026cc33437c40e7" is_pre_election: true
I20260926 01:05:11.934038 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.930564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38603 (local address 127.0.0.1:54150)
0926 01:05:11.930991 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.931001 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:11.931052 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:11.931255 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:11.931256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:11.931258 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:11.931317 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:11.931319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:11.932109 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:11.932110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:11.932926 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:11.932928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:11.933457 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:11.933458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:11.933511 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:11.933514 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:11.933558 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:05:11.933969 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":2,"thread_start_us":35,"threads_started":1}
I20260926 01:05:11.934041 7087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1edf6b838f9d4255bb2ee0aac9a9913a in term 1.
I20260926 01:05:11.934310 7249 leader_election.cc:302] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1edf6b838f9d4255bb2ee0aac9a9913a, d7cda31285d14786b026cc33437c40e7; no voters: b0573a39c6e042d19458b0849f990d9b
I20260926 01:05:11.934438 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7cda31285d14786b026cc33437c40e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" is_pre_election: true
I20260926 01:05:11.934453 7787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:05:11.934535 7273 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7cda31285d14786b026cc33437c40e7 in term 1.
I20260926 01:05:11.934568 7787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 FOLLOWER]: Starting leader election (detected failure of leader b0573a39c6e042d19458b0849f990d9b)
I20260926 01:05:11.934587 7787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:05:11.934693 7063 leader_election.cc:302] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1edf6b838f9d4255bb2ee0aac9a9913a, d7cda31285d14786b026cc33437c40e7; no voters: b0573a39c6e042d19458b0849f990d9b
I20260926 01:05:11.934808 7786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:05:11.934840 7786 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b0573a39c6e042d19458b0849f990d9b)
I20260926 01:05:11.934859 7786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:05:11.935400 7787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } }
I20260926 01:05:11.935513 7787 leader_election.cc:288] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [CANDIDATE]: Term 2 election: Requested vote from peers b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137), d7cda31285d14786b026cc33437c40e7 (127.0.98.61:43727)
I20260926 01:05:11.935472 7786 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } }
I20260926 01:05:11.935608 7786 leader_election.cc:288] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 2 election: Requested vote from peers b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137), 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60:38603)
I20260926 01:05:11.935763 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.935631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58260)
0926 01:05:11.935709 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.935737 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 01:05:11.935750 7087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d7cda31285d14786b026cc33437c40e7"
I20260926 01:05:11.935752 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7cda31285d14786b026cc33437c40e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a"
I20260926 01:05:11.935842 7087 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1edf6b838f9d4255bb2ee0aac9a9913a in current term 2: Already voted for candidate d7cda31285d14786b026cc33437c40e7 in this term.
I20260926 01:05:11.935860 7273 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cda31285d14786b026cc33437c40e7 in current term 2: Already voted for candidate 1edf6b838f9d4255bb2ee0aac9a9913a in this term.
I20260926 01:05:11.936041 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.935690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58266)
0926 01:05:11.935765 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.935794 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 01:05:11.936064 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.935980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58268)
0926 01:05:11.936014 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.936042 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 01:05:11.936157 7249 leader_election.cc:334] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
I20260926 01:05:11.936231 7249 leader_election.cc:302] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1edf6b838f9d4255bb2ee0aac9a9913a; no voters: b0573a39c6e042d19458b0849f990d9b, d7cda31285d14786b026cc33437c40e7
I20260926 01:05:11.936350 7787 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 01:05:11.936549 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:11.936298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58282)
0926 01:05:11.936496 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:11.936527 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260926 01:05:11.936625 7062 leader_election.cc:334] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
I20260926 01:05:11.936664 7062 leader_election.cc:302] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cda31285d14786b026cc33437c40e7; no voters: 1edf6b838f9d4255bb2ee0aac9a9913a, b0573a39c6e042d19458b0849f990d9b
I20260926 01:05:11.936753 7786 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 01:05:12.427037 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:12.426820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:36035)
0926 01:05:12.426939 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:12.426984 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":44,"threads_started":1}
I20260926 01:05:12.427054 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:12.426762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:41645)
0926 01:05:12.426972 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:12.427007 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260926 01:05:12.427860 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:12.427649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:37499)
0926 01:05:12.427773 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:12.427818 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260926 01:05:13.427740 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.427479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:49747)
0926 01:05:13.427633 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.427683 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
I20260926 01:05:13.427868 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.427642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:58573)
0926 01:05:13.427787 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.427821 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":41,"threads_started":1}
I20260926 01:05:13.428388 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.428246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:45213)
0926 01:05:13.428329 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.428353 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":39,"threads_started":1}
I20260926 01:05:13.559545 7802 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:13.559649 7802 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } }
I20260926 01:05:13.559811 7802 leader_election.cc:288] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137), 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60:38603)
I20260926 01:05:13.560062 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7cda31285d14786b026cc33437c40e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" is_pre_election: true
I20260926 01:05:13.560155 7273 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7cda31285d14786b026cc33437c40e7 in term 2.
I20260926 01:05:13.560163 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.559970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58508)
0926 01:05:13.560085 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.560122 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260926 01:05:13.560288 7063 leader_election.cc:302] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1edf6b838f9d4255bb2ee0aac9a9913a, d7cda31285d14786b026cc33437c40e7; no voters:
I20260926 01:05:13.560379 7802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 01:05:13.560457 7802 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:13.560478 7802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:05:13.560652 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.560538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58516)
0926 01:05:13.560571 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.560629 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 01:05:13.560729 7062 leader_election.cc:334] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
I20260926 01:05:13.561254 7802 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } }
I20260926 01:05:13.561367 7802 leader_election.cc:288] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 3 election: Requested vote from peers b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137), 1edf6b838f9d4255bb2ee0aac9a9913a (127.0.98.60:38603)
I20260926 01:05:13.561532 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7cda31285d14786b026cc33437c40e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a"
I20260926 01:05:13.561571 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.561468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58518)
0926 01:05:13.561517 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.561548 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 01:05:13.561621 7273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:05:13.561949 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.561878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58520)
0926 01:05:13.561903 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.561929 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 01:05:13.562031 7062 leader_election.cc:334] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
I20260926 01:05:13.562367 7273 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cda31285d14786b026cc33437c40e7 in term 3.
I20260926 01:05:13.562557 7063 leader_election.cc:302] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1edf6b838f9d4255bb2ee0aac9a9913a, d7cda31285d14786b026cc33437c40e7; no voters: b0573a39c6e042d19458b0849f990d9b
I20260926 01:05:13.562656 7802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 01:05:13.562781 7802 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [term 3 LEADER]: Becoming Leader. State: Replica: d7cda31285d14786b026cc33437c40e7, State: Running, Role: LEADER
I20260926 01:05:13.562929 7802 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } }
I20260926 01:05:13.563287 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7cda31285d14786b026cc33437c40e7. Latest consensus state: current_term: 3 leader_uuid: "d7cda31285d14786b026cc33437c40e7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:13.563387 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:13.563694 7807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:13.564109 7807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:13.564167 7807 catalog_manager.cc:1259] Loaded cluster ID: e6ab4cb480ae405ab602f8b999ba1c19
I20260926 01:05:13.564181 7807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:13.564359 7807 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:13.564487 7807 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Loaded TSK: 0
I20260926 01:05:13.564632 7807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:05:13.611482 7077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:05:13.613029 7077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:05:13.614086 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.613956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58538)
0926 01:05:13.614012 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.614048 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 01:05:13.614084 7273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [term 3 FOLLOWER]: Refusing update from remote peer d7cda31285d14786b026cc33437c40e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 01:05:13.614301 7805 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 01:05:13.614424 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.614319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:58550)
0926 01:05:13.614359 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.614400 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 01:05:13.614496 7062 consensus_peers.cc:487] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 -> Peer b0573a39c6e042d19458b0849f990d9b (127.0.98.62:35137): Couldn't send request to peer b0573a39c6e042d19458b0849f990d9b. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:05:13.614962 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7cda31285d14786b026cc33437c40e7. Latest consensus state: current_term: 3 leader_uuid: "d7cda31285d14786b026cc33437c40e7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:13.615056 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:05:13.615607 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d7cda31285d14786b026cc33437c40e7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:13.615700 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edf6b838f9d4255bb2ee0aac9a9913a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:05:13.615767 7802 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d7cda31285d14786b026cc33437c40e7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b0573a39c6e042d19458b0849f990d9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } peers { permanent_uuid: "d7cda31285d14786b026cc33437c40e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43727 } attrs { promote: false } } peers { permanent_uuid: "1edf6b838f9d4255bb2ee0aac9a9913a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38603 } attrs { promote: false } } }
I20260926 01:05:13.615844 7802 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:13.620551 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.617300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.0.98.1:43003)
0926 01:05:13.617413 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:13.617415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:13.617427 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:13.617456 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:13.617457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:13.617469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:13.617538 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:13.617812 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.618671 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.619623 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.620184 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.620258 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.620272 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:13.620427 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 01:05:13.620467 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260926 01:05:13.620575 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.617132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43003 (local address 127.0.0.1:56736)
0926 01:05:13.617171 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.617181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:13.617224 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:13.617555 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:13.617556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:13.617557 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:13.617629 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:13.617631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.618775 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.618776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:13.619525 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:13.619527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.620271 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.620272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:13.620341 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.620352 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 01:05:13.620417 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 01:05:13.620464 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 01:05:13.620797 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.617225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39833 (local address 127.0.0.1:36266)
0926 01:05:13.617544 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.617551 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:13.617589 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:13.617767 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:13.617768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:13.617769 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:13.617811 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:13.617812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.618752 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.618752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:13.619817 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:13.619819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.620579 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.620580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:13.620634 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.620637 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:13.620682 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:05:13.620723 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":32,"thread_start_us":87,"threads_started":1}
I20260926 01:05:13.620999 7377 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbf4cd4cc4a4df4b01d90bc32acd409 (DEFAULT_TABLE table=test-table [id=18770de117dc4227aae150f1a34a5ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:13.621076 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.617323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.0.98.2:39833)
0926 01:05:13.617470 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:13.617472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:13.617634 (+ 162us) server_negotiation.cc:407] Connection header received
0926 01:05:13.617662 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:13.617663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:13.617672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:13.617727 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:13.617869 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.618662 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.619899 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.620484 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.620538 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.620550 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:13.620960 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 01:05:13.621005 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260926 01:05:13.621131 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.617225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40797 (local address 127.0.0.1:37228)
0926 01:05:13.617413 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.617422 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:13.617476 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:13.617635 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:13.617636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:13.617638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:13.617665 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:13.617666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.618423 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.618424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:13.619481 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:13.619483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.620337 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.620338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:13.620407 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.620411 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:13.620468 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 01:05:13.620515 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
I20260926 01:05:13.621145 7377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbf4cd4cc4a4df4b01d90bc32acd409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:13.621250 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbf4cd4cc4a4df4b01d90bc32acd409 (DEFAULT_TABLE table=test-table [id=18770de117dc4227aae150f1a34a5ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:13.621294 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.617427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.98.3:40797)
0926 01:05:13.617520 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:13.617521 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:13.617529 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:13.617553 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:13.617554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:13.617565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:13.617634 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:13.617723 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.618326 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.619582 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.620232 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.620296 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.620308 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:13.620603 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 01:05:13.621228 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":44,"threads_started":1}
I20260926 01:05:13.621377 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbf4cd4cc4a4df4b01d90bc32acd409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:13.622632 7819 tablet_bootstrap.cc:492] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77: Bootstrap starting.
I20260926 01:05:13.622649 7820 tablet_bootstrap.cc:492] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00: Bootstrap starting.
I20260926 01:05:13.622911 7650 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbf4cd4cc4a4df4b01d90bc32acd409 (DEFAULT_TABLE table=test-table [id=18770de117dc4227aae150f1a34a5ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:13.623199 7650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbf4cd4cc4a4df4b01d90bc32acd409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:13.623275 7820 tablet_bootstrap.cc:654] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:13.623812 7820 tablet_bootstrap.cc:492] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00: No bootstrap required, opened a new log
I20260926 01:05:13.623858 7820 ts_tablet_manager.cc:1397] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:13.624121 7820 raft_consensus.cc:348] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260926 01:05:13.624193 7820 raft_consensus.cc:374] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:13.624217 7820 raft_consensus.cc:729] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478ce1ac7e994e7db5035b881a7bfa00, State: Initialized, Role: FOLLOWER
I20260926 01:05:13.624258 7820 consensus_queue.cc:260] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260926 01:05:13.624425 7820 ts_tablet_manager.cc:1428] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:13.624725 7819 tablet_bootstrap.cc:654] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:13.624755 7822 tablet_bootstrap.cc:492] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb: Bootstrap starting.
I20260926 01:05:13.625326 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.624784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:49259)
0926 01:05:13.625249 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.625293 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324}
I20260926 01:05:13.625506 7445 heartbeater.cc:503] Master 127.0.98.61:43727 was elected leader, sending a full tablet report...
I20260926 01:05:13.625571 7822 tablet_bootstrap.cc:654] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:13.625728 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.625619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:38355)
0926 01:05:13.625676 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.625704 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 01:05:13.625996 7819 tablet_bootstrap.cc:492] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77: No bootstrap required, opened a new log
I20260926 01:05:13.626149 7819 ts_tablet_manager.cc:1397] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 01:05:13.626295 7819 raft_consensus.cc:348] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260926 01:05:13.626356 7822 tablet_bootstrap.cc:492] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb: No bootstrap required, opened a new log
I20260926 01:05:13.626412 7819 raft_consensus.cc:374] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:13.626430 7822 ts_tablet_manager.cc:1397] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:13.626438 7819 raft_consensus.cc:729] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e08a68241a46fb9685683facdbaf77, State: Initialized, Role: FOLLOWER
I20260926 01:05:13.626503 7819 consensus_queue.cc:260] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260926 01:05:13.626529 7822 raft_consensus.cc:348] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260926 01:05:13.626593 7822 raft_consensus.cc:374] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:13.626610 7822 raft_consensus.cc:729] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc5dc1a23274a64aa4b90e7f583dbeb, State: Initialized, Role: FOLLOWER
I20260926 01:05:13.626662 7822 consensus_queue.cc:260] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260926 01:05:13.626735 7819 ts_tablet_manager.cc:1428] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:13.626832 7822 ts_tablet_manager.cc:1428] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:13.627067 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.626944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:54731)
0926 01:05:13.627005 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.627040 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 01:05:13.627293 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.627202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:47569)
0926 01:05:13.627239 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.627269 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 01:05:13.627506 7718 heartbeater.cc:503] Master 127.0.98.61:43727 was elected leader, sending a full tablet report...
I20260926 01:05:13.627789 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.627702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:49917)
0926 01:05:13.627737 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.627764 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 01:05:13.627801 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.627712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:58069)
0926 01:05:13.627750 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.627774 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 01:05:13.812812 7827 raft_consensus.cc:482] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:13.812939 7827 raft_consensus.cc:504] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260926 01:05:13.813306 7827 leader_election.cc:288] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 478ce1ac7e994e7db5035b881a7bfa00 (127.0.98.1:43003), 32e08a68241a46fb9685683facdbaf77 (127.0.98.2:39833)
I20260926 01:05:13.816468 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.813588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57001 (local address 127.0.98.1:43003)
0926 01:05:13.813641 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:13.813642 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:13.813653 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:13.813736 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:13.813737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:13.813747 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:13.813798 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:13.814055 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.814608 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.815680 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.816095 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.816163 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.816175 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:13.816351 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 01:05:13.816392 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:05:13.816505 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.813631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49343 (local address 127.0.98.2:39833)
0926 01:05:13.813673 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:13.813684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:13.813824 (+ 140us) server_negotiation.cc:407] Connection header received
0926 01:05:13.813870 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:13.813871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:13.813883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:13.813945 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:13.814202 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.814793 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.815753 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.816138 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.816222 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.816235 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:13.816396 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 01:05:13.816449 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:05:13.816468 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.813510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43003 (local address 127.0.98.3:57001)
0926 01:05:13.813594 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.813606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:13.813664 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:13.813818 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:13.813819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:13.813822 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:13.813913 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:13.813915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.814686 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.814687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:13.815563 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:13.815566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.816199 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.816201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:13.816277 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.816282 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:05:13.816325 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 01:05:13.816362 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 01:05:13.816573 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.813552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39833 (local address 127.0.98.3:49343)
0926 01:05:13.813759 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.813768 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:13.813813 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:13.814016 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:13.814017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:13.814020 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:13.814079 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:13.814081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.814895 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.814896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:13.815648 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:13.815651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.816264 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.816265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:13.816317 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.816321 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:13.816373 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 01:05:13.816413 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":114,"threads_started":1}
I20260926 01:05:13.816740 7397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbf4cd4cc4a4df4b01d90bc32acd409" candidate_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478ce1ac7e994e7db5035b881a7bfa00" is_pre_election: true
I20260926 01:05:13.816831 7397 raft_consensus.cc:2454] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc5dc1a23274a64aa4b90e7f583dbeb in term 0.
I20260926 01:05:13.817030 7605 leader_election.cc:302] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 478ce1ac7e994e7db5035b881a7bfa00, edc5dc1a23274a64aa4b90e7f583dbeb; no voters:
I20260926 01:05:13.817003 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbf4cd4cc4a4df4b01d90bc32acd409" candidate_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e08a68241a46fb9685683facdbaf77" is_pre_election: true
I20260926 01:05:13.817123 7534 raft_consensus.cc:2454] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc5dc1a23274a64aa4b90e7f583dbeb in term 0.
I20260926 01:05:13.817165 7827 raft_consensus.cc:2781] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:05:13.817281 7827 raft_consensus.cc:482] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:13.817315 7827 raft_consensus.cc:3037] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:13.817988 7827 raft_consensus.cc:504] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260926 01:05:13.818114 7827 leader_election.cc:288] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [CANDIDATE]: Term 1 election: Requested vote from peers 478ce1ac7e994e7db5035b881a7bfa00 (127.0.98.1:43003), 32e08a68241a46fb9685683facdbaf77 (127.0.98.2:39833)
I20260926 01:05:13.818282 7397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbf4cd4cc4a4df4b01d90bc32acd409" candidate_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478ce1ac7e994e7db5035b881a7bfa00"
I20260926 01:05:13.818313 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbf4cd4cc4a4df4b01d90bc32acd409" candidate_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e08a68241a46fb9685683facdbaf77"
I20260926 01:05:13.818374 7397 raft_consensus.cc:3037] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:13.818374 7534 raft_consensus.cc:3037] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:13.819201 7397 raft_consensus.cc:2454] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc5dc1a23274a64aa4b90e7f583dbeb in term 1.
I20260926 01:05:13.819228 7534 raft_consensus.cc:2454] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc5dc1a23274a64aa4b90e7f583dbeb in term 1.
I20260926 01:05:13.819446 7606 leader_election.cc:302] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e08a68241a46fb9685683facdbaf77, edc5dc1a23274a64aa4b90e7f583dbeb; no voters:
I20260926 01:05:13.819617 7827 raft_consensus.cc:2781] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:13.819806 7827 raft_consensus.cc:686] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 1 LEADER]: Becoming Leader. State: Replica: edc5dc1a23274a64aa4b90e7f583dbeb, State: Running, Role: LEADER
I20260926 01:05:13.819991 7827 consensus_queue.cc:237] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260926 01:05:13.820206 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.820062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:55561)
0926 01:05:13.820131 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.820172 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 01:05:13.820737 7076 catalog_manager.cc:5495] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb reported cstate change: term changed from 0 to 1, leader changed from <none> to edc5dc1a23274a64aa4b90e7f583dbeb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc5dc1a23274a64aa4b90e7f583dbeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } health_report { overall_health: UNKNOWN } } }
I20260926 01:05:13.850304 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.847605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.0.98.3:40797)
0926 01:05:13.847693 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:13.847695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:13.847704 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:13.847737 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:13.847738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:13.847758 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:13.847817 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:13.847970 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.848482 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.849347 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:13.849597 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:13.849663 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:13.849832 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:13.849937 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:13.849991 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:13.850182 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 01:05:13.850212 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 01:05:13.851665 7397 raft_consensus.cc:1261] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 478ce1ac7e994e7db5035b881a7bfa00 [term 1 FOLLOWER]: Refusing update from remote peer edc5dc1a23274a64aa4b90e7f583dbeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:13.851735 7534 raft_consensus.cc:1261] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77 [term 1 FOLLOWER]: Refusing update from remote peer edc5dc1a23274a64aa4b90e7f583dbeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:13.851944 7827 consensus_queue.cc:1035] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "478ce1ac7e994e7db5035b881a7bfa00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:13.852092 7829 consensus_queue.cc:1035] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e08a68241a46fb9685683facdbaf77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:13.852689 7582 heartbeater.cc:503] Master 127.0.98.61:43727 was elected leader, sending a full tablet report...
I20260926 01:05:13.853006 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.852647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:52561)
0926 01:05:13.852929 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.852969 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253}
I20260926 01:05:13.853087 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.852989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:34663)
0926 01:05:13.853031 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.853058 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 01:05:13.855135 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.855007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:54693)
0926 01:05:13.855072 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.855101 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 01:05:13.855265 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.855149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:36669)
0926 01:05:13.855202 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.855231 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:05:13.855901 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.854515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:37111)
0926 01:05:13.855834 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.855867 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1300}
I20260926 01:05:13.856303 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.856177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:56567)
0926 01:05:13.856239 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.856275 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 01:05:13.856725 7077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:13.856812 7077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:13.858120 7077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7cda31285d14786b026cc33437c40e7: Sending DeleteTablet for 3 replicas of tablet 8cbf4cd4cc4a4df4b01d90bc32acd409
I20260926 01:05:13.858446 7377 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbf4cd4cc4a4df4b01d90bc32acd409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:13 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56736
I20260926 01:05:13.858465 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbf4cd4cc4a4df4b01d90bc32acd409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:13 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36266
I20260926 01:05:13.858796 7843 tablet_replica.cc:330] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77: stopping tablet replica
I20260926 01:05:13.858853 7843 raft_consensus.cc:2229] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:13.858773 7650 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbf4cd4cc4a4df4b01d90bc32acd409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:13 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37228
I20260926 01:05:13.858903 7843 raft_consensus.cc:2258] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:13.859048 7845 tablet_replica.cc:330] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb: stopping tablet replica
I20260926 01:05:13.859135 7845 raft_consensus.cc:2229] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:05:13.859272 7843 ts_tablet_manager.cc:1905] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:13.859378 7845 raft_consensus.cc:2258] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:13.859683 7845 ts_tablet_manager.cc:1905] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:13.859154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 7225
I20260926 01:05:13.861486 7843 ts_tablet_manager.cc:1918] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:13.861486 7845 ts_tablet_manager.cc:1918] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:13.861588 7843 log.cc:1196] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/8cbf4cd4cc4a4df4b01d90bc32acd409
I20260926 01:05:13.861644 7845 log.cc:1196] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/8cbf4cd4cc4a4df4b01d90bc32acd409
I20260926 01:05:13.861845 7843 ts_tablet_manager.cc:1939] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P 32e08a68241a46fb9685683facdbaf77: Deleting consensus metadata
I20260926 01:05:13.861871 7845 ts_tablet_manager.cc:1939] T 8cbf4cd4cc4a4df4b01d90bc32acd409 P edc5dc1a23274a64aa4b90e7f583dbeb: Deleting consensus metadata
I20260926 01:05:13.862233 7062 catalog_manager.cc:4841] TS edc5dc1a23274a64aa4b90e7f583dbeb (127.0.98.3:40797): tablet 8cbf4cd4cc4a4df4b01d90bc32acd409 (table test-table [id=18770de117dc4227aae150f1a34a5ced]) successfully deleted
I20260926 01:05:13.862242 7064 catalog_manager.cc:4841] TS 32e08a68241a46fb9685683facdbaf77 (127.0.98.2:39833): tablet 8cbf4cd4cc4a4df4b01d90bc32acd409 (table test-table [id=18770de117dc4227aae150f1a34a5ced]) successfully deleted
I20260926 01:05:13.865302 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:13.865171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43003 (local address 127.0.0.1:56750)
0926 01:05:13.865233 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:13.865264 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 01:05:13.865438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 7451
W20260926 01:05:13.865492 7063 catalog_manager.cc:4584] TS 478ce1ac7e994e7db5035b881a7bfa00 (127.0.98.1:43003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8cbf4cd4cc4a4df4b01d90bc32acd409: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43003: connect: Connection refused (error 111)
I20260926 01:05:13.872025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 7587
I20260926 01:05:13.876793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 6944
I20260926 01:05:13.881263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 7131
2026-09-26T01:05:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6728 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:05:13 dist-test-slave-l73v krb5kdc[7848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:05:13 dist-test-slave-l73v krb5kdc[7848](info): set up 2 sockets
Sep 26 01:05:13 dist-test-slave-l73v krb5kdc[7848](info): commencing operation
krb5kdc: starting...
W20260926 01:05:14.854883 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:05:14 dist-test-slave-l73v krb5kdc[7848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 01:05:14.986289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43161
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:15.074050 7861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:15.074311 7861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:15.074337 7861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:15.075686 7861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:15.075747 7861 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 01:05:15.075759 7861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:15.075771 7861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:15.075783 7861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:15.075793 7861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:15.077395 7861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43161
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:15.077641 7861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:15.077834 7861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:15.083026 7867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:15.083045 7866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:15.083217 7869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:15.083447 7861 server_base.cc:1034] running on GCE node
I20260926 01:05:15.083695 7861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:15.083999 7861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:15.085160 7861 hybrid_clock.cc:648] HybridClock initialized: now 1790384715085140 us; error 37 us; skew 500 ppm
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:15.087847 7861 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 01:05:15.088415 7861 webserver.cc:460] Webserver started at http://127.0.98.62:41987/ using document root <none> and password file <none>
I20260926 01:05:15.088646 7861 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:15.088698 7861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:15.088833 7861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:15.089890 7861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "beac26dde2184e65b6278720aeede609"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "4d5b38ef74808f4f89d3f9cb57dfea06"
server_key_iv: "d62e93564e4282535e1c374e6e892521"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.090217 7861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "beac26dde2184e65b6278720aeede609"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "4d5b38ef74808f4f89d3f9cb57dfea06"
server_key_iv: "d62e93564e4282535e1c374e6e892521"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.091661 7861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:15.092721 7876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:15.093118 7861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:15.093232 7861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "beac26dde2184e65b6278720aeede609"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "4d5b38ef74808f4f89d3f9cb57dfea06"
server_key_iv: "d62e93564e4282535e1c374e6e892521"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.093313 7861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:15.130151 7861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:15.132512 7861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:15.132678 7861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:15.137835 7861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43161
I20260926 01:05:15.137887 7938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43161 every 8 connection(s)
I20260926 01:05:15.138244 7861 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:15.138921 7939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:15.141268 7939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609: Bootstrap starting.
I20260926 01:05:15.141841 7939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:15.142138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 7861
I20260926 01:05:15.142230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:15.142304 7939 log.cc:824] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:15.142452 392 external_mini_cluster.cc:1428] Setting key 677112c55eaaa565a3f9d3e17df5c02c
I20260926 01:05:15.143414 7939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609: No bootstrap required, opened a new log
I20260926 01:05:15.145934 7939 raft_consensus.cc:348] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beac26dde2184e65b6278720aeede609" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43161 } }
I20260926 01:05:15.146248 7939 raft_consensus.cc:374] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:15.146301 7939 raft_consensus.cc:729] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beac26dde2184e65b6278720aeede609, State: Initialized, Role: FOLLOWER
I20260926 01:05:15.146497 7939 consensus_queue.cc:260] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beac26dde2184e65b6278720aeede609" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43161 } }
I20260926 01:05:15.146596 7939 raft_consensus.cc:388] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:15.146628 7939 raft_consensus.cc:482] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:15.146734 7939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:15.147763 7939 raft_consensus.cc:504] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beac26dde2184e65b6278720aeede609" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43161 } }
I20260926 01:05:15.147891 7939 leader_election.cc:302] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beac26dde2184e65b6278720aeede609; no voters:
I20260926 01:05:15.148088 7939 leader_election.cc:288] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:15.148187 7944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:15.148407 7944 raft_consensus.cc:686] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [term 1 LEADER]: Becoming Leader. State: Replica: beac26dde2184e65b6278720aeede609, State: Running, Role: LEADER
I20260926 01:05:15.148546 7944 consensus_queue.cc:237] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beac26dde2184e65b6278720aeede609" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43161 } }
I20260926 01:05:15.148967 7945 sys_catalog.cc:455] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beac26dde2184e65b6278720aeede609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beac26dde2184e65b6278720aeede609" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43161 } } }
I20260926 01:05:15.149053 7945 sys_catalog.cc:458] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:15.149271 7948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:15.149296 7946 sys_catalog.cc:455] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beac26dde2184e65b6278720aeede609. Latest consensus state: current_term: 1 leader_uuid: "beac26dde2184e65b6278720aeede609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beac26dde2184e65b6278720aeede609" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43161 } } }
I20260926 01:05:15.149600 7946 sys_catalog.cc:458] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:15.149863 7948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:15.148551 7939 sys_catalog.cc:564] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:15.151635 7948 catalog_manager.cc:1347] Generated new cluster ID: 075a43788e624c5ba893effbc26186a8
I20260926 01:05:15.151695 7948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:05:15.158080 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.143056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.0.98.62:43161)
0926 01:05:15.143237 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.143247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.146785 (+ 3538us) server_negotiation.cc:407] Connection header received
0926 01:05:15.146999 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.147008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.147217 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.147412 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:15.148694 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.150062 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.151237 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.151873 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.151989 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.154293 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:15.154299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:15.154299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:15.154317 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:15.156432 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.156828 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.156832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.156832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.156952 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.157143 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.157144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.157144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.157428 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:15.157589 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.157697 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.157852 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260926 01:05:15.178108 7948 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:15.178817 7948 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:15.185701 7948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609: Generated new TSK 0
I20260926 01:05:15.186025 7948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 01:05:15.223098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43161
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:15.341708 7965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:15.341979 7965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:15.342006 7965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:15.343331 7965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:15.343384 7965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:15.343406 7965 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:05:15.343423 7965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:15.345011 7965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:15.345227 7965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:15.345566 7965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:15.350977 7971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:15.350977 7973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:15.351032 7970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:15.351240 7965 server_base.cc:1034] running on GCE node
I20260926 01:05:15.351547 7965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:15.351822 7965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:15.352994 7965 hybrid_clock.cc:648] HybridClock initialized: now 1790384715352970 us; error 41 us; skew 500 ppm
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:15.355728 7965 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 01:05:15.356248 7965 webserver.cc:460] Webserver started at http://127.0.98.1:36647/ using document root <none> and password file <none>
I20260926 01:05:15.356495 7965 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:15.356551 7965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:15.356684 7965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:15.357780 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "60f2c39fca3144b3903c5ab0d2929d1d"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "d66201c098a66a069bf9dd8201310510"
server_key_iv: "47b04e13c510949902e131fcaa4c0153"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.358111 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "60f2c39fca3144b3903c5ab0d2929d1d"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "d66201c098a66a069bf9dd8201310510"
server_key_iv: "47b04e13c510949902e131fcaa4c0153"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.359613 7965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:15.360474 7980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:15.360733 7965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:15.360836 7965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "60f2c39fca3144b3903c5ab0d2929d1d"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "d66201c098a66a069bf9dd8201310510"
server_key_iv: "47b04e13c510949902e131fcaa4c0153"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.360981 7965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:15.385308 7965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:15.387431 7965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:15.387583 7965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:15.388176 7965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:15.388229 7965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:15.388258 7965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:15.388274 7965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:05:15.395121 7965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36881
I20260926 01:05:15.395191 8095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36881 every 8 connection(s)
I20260926 01:05:15.395509 7965 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:05:15.397851 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.388660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59967 (local address 127.0.98.62:43161)
0926 01:05:15.388732 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.388734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.388936 (+ 202us) server_negotiation.cc:407] Connection header received
0926 01:05:15.389696 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.389698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.389721 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.389800 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:15.390712 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.391214 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.392293 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.392488 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.392546 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.395090 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:15.395092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:15.395093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:15.395106 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:15.396617 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.397068 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.397069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.397070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.397129 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.397306 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.397307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.397307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.397442 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:15.397495 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.397709 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.397787 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 01:05:15.398008 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.388624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43161 (local address 127.0.98.1:59967)
0926 01:05:15.388780 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.388831 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.389529 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.389926 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.389931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.390330 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:15.390608 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.390614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.391314 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.391315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.392196 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.392199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.392579 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.392580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.392701 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.393226 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:15.393236 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:15.394955 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:15.396748 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.396754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.396757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.397008 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.397180 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.397180 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.397180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.397269 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.397485 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:15.397487 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:15.397601 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.397678 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.397808 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260926 01:05:15.398504 8096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43161
I20260926 01:05:15.398586 8096 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:15.398820 8096 heartbeater.cc:511] Master 127.0.98.62:43161 requested a full tablet report, sending...
I20260926 01:05:15.399325 7892 ts_manager.cc:194] Registered new tserver with Master: 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881)
I20260926 01:05:15.400200 7892 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59967
I20260926 01:05:15.400633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 7965
I20260926 01:05:15.400751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:05:15.400954 392 external_mini_cluster.cc:1428] Setting key fc482beab28c402cb1d3f7a82b1b2f3a
I20260926 01:05:15.405645 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.399431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43161 (local address 127.0.98.1:45761)
0926 01:05:15.399556 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.399569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.399629 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.399735 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.399736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.399936 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:15.399996 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.399997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.400886 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.400887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.401502 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.401505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.401925 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.401926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.401990 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.402480 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:15.402485 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:15.402872 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:15.404554 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.404556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.404556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.404775 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.405001 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.405002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.405002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.405067 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.405347 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:15.405348 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:15.405399 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.405456 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.405558 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260926 01:05:15.405985 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.399490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45761 (local address 127.0.98.62:43161)
0926 01:05:15.399528 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.399530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.399615 (+ 85us) server_negotiation.cc:407] Connection header received
0926 01:05:15.399680 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.399680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.399688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.399729 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:15.400092 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.400783 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.401590 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.401834 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.401902 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.402980 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:15.402982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:15.402982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:15.402989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:15.404462 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.404848 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.404849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.404849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.404909 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.405132 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.405133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.405140 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.405281 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:15.405349 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.405539 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.405637 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 01:05:15.431555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43161
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:15.513144 8105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:15.513381 8105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:15.513406 8105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:15.514647 8105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:15.514698 8105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:15.514717 8105 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:05:15.514734 8105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:15.516271 8105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:15.516520 8105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:15.516717 8105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:15.521997 8110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:15.522017 8113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:15.522490 8111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:15.522974 8105 server_base.cc:1034] running on GCE node
I20260926 01:05:15.523243 8105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:15.523515 8105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:15.524659 8105 hybrid_clock.cc:648] HybridClock initialized: now 1790384715524640 us; error 39 us; skew 500 ppm
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:15.527226 8105 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 01:05:15.527658 8105 webserver.cc:460] Webserver started at http://127.0.98.2:34957/ using document root <none> and password file <none>
I20260926 01:05:15.527834 8105 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:15.527876 8105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:15.527971 8105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:15.528982 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "a170d255d1084a6a8a5373bd97cd0510"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "662b10f90ecea260d21afb1b231faacf"
server_key_iv: "e6d8cc64256eaff6216c30bf53b941ec"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.529274 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a170d255d1084a6a8a5373bd97cd0510"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "662b10f90ecea260d21afb1b231faacf"
server_key_iv: "e6d8cc64256eaff6216c30bf53b941ec"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.530678 8105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:15.531540 8120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:15.531747 8105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:15.531860 8105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "a170d255d1084a6a8a5373bd97cd0510"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "662b10f90ecea260d21afb1b231faacf"
server_key_iv: "e6d8cc64256eaff6216c30bf53b941ec"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.531929 8105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:15.552026 8105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:15.554253 8105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:15.554373 8105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:15.555056 8105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:15.555104 8105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:15.555141 8105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:15.555171 8105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:15.561709 8105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42899
I20260926 01:05:15.562126 8105 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:05:15.562353 8235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42899 every 8 connection(s)
I20260926 01:05:15.566500 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.555498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54539 (local address 127.0.98.62:43161)
0926 01:05:15.555542 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.555545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.555766 (+ 221us) server_negotiation.cc:407] Connection header received
0926 01:05:15.556299 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.556301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.556322 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.556404 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:15.557275 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.558046 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.559656 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.559961 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.560044 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.563642 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:15.563644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:15.563644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:15.563655 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:15.564947 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.565450 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.565451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.565452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.565512 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.565749 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.565751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.565751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.565888 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:15.565957 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.566304 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.566433 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:05:15.566625 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.555492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43161 (local address 127.0.98.2:54539)
0926 01:05:15.555628 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.555665 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.556195 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.556479 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.556483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.556836 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:15.557147 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.557154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.558153 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.558154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.559552 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.559555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.560048 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.560048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.560156 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.560748 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:15.560766 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:15.563507 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:15.565083 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.565092 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.565095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.565358 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.565570 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.565571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.565571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.565667 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.565949 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:15.565952 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:15.566082 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.566176 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.566373 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260926 01:05:15.566915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 8105
I20260926 01:05:15.567035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:05:15.567270 392 external_mini_cluster.cc:1428] Setting key 4c013ad324e4884af830d131093580e5
I20260926 01:05:15.567297 8236 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43161
I20260926 01:05:15.567382 8236 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:15.567581 8236 heartbeater.cc:511] Master 127.0.98.62:43161 requested a full tablet report, sending...
I20260926 01:05:15.567919 7892 ts_manager.cc:194] Registered new tserver with Master: a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899)
I20260926 01:05:15.568338 7892 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54539
I20260926 01:05:15.574086 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.568451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36313 (local address 127.0.98.62:43161)
0926 01:05:15.568538 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.568540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.568551 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:15.568570 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.568571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.568579 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.568625 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:15.568988 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.569680 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.570412 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.570608 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.570666 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.571603 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:15.571604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:15.571605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:15.571612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:15.572965 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.573312 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.573313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.573313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.573371 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.573542 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.573542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.573543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.573699 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:15.573758 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.573907 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.573986 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 01:05:15.574100 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.568338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43161 (local address 127.0.98.2:36313)
0926 01:05:15.568468 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.568479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.568531 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.568631 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.568631 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.568821 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:15.568894 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.568896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.569771 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.569772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.570291 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.570293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.570675 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.570676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.570742 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.571191 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:15.571197 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:15.571518 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:15.573069 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.573070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.573070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.573243 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.573429 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.573430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.573430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.573489 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.573766 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:15.573767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:15.573817 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.573875 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.573987 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 01:05:15.594643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43161
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:15.676311 8245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:15.676626 8245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:15.676680 8245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:15.678524 8245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:15.678591 8245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:15.678622 8245 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 01:05:15.678655 8245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:15.681173 8245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:15.681483 8245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:15.681758 8245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:05:15.688024 8245 server_base.cc:1034] running on GCE node
W20260926 01:05:15.687976 8251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:15.688125 8250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:15.688247 8253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:15.688470 8245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:15.688695 8245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:15.689860 8245 hybrid_clock.cc:648] HybridClock initialized: now 1790384715689834 us; error 35 us; skew 500 ppm
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:15.693423 8245 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 01:05:15.693887 8245 webserver.cc:460] Webserver started at http://127.0.98.3:34743/ using document root <none> and password file <none>
I20260926 01:05:15.694149 8245 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:15.694209 8245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:15.694342 8245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:15.695688 8245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "927c3fa6a15943e98a45fe6e158dfd89"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "c4041d860f25dfd9cacd1e0e0c61de16"
server_key_iv: "1f30378e81286847065c2d0b244550a3"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.696110 8245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "927c3fa6a15943e98a45fe6e158dfd89"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "c4041d860f25dfd9cacd1e0e0c61de16"
server_key_iv: "1f30378e81286847065c2d0b244550a3"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.697826 8245 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 01:05:15.698649 8260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:15.699720 8245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:15.699826 8245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "927c3fa6a15943e98a45fe6e158dfd89"
format_stamp: "Formatted at 2026-09-26 01:05:15 on dist-test-slave-l73v"
server_key: "c4041d860f25dfd9cacd1e0e0c61de16"
server_key_iv: "1f30378e81286847065c2d0b244550a3"
server_key_version: "encryptionkey@0"
I20260926 01:05:15.699903 8245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:15.741683 8245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:15.744460 8245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:15.744623 8245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:15.745185 8245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:15.745231 8245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:15.745263 8245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:15.745280 8245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:15.752743 8245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46343
I20260926 01:05:15.752899 8375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46343 every 8 connection(s)
I20260926 01:05:15.753181 8245 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:05:15.756809 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.745741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41019 (local address 127.0.98.62:43161)
0926 01:05:15.745821 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.745824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.746024 (+ 200us) server_negotiation.cc:407] Connection header received
0926 01:05:15.746590 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.746592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.746604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.746655 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:15.748142 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.749122 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.750822 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.751275 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.751363 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.753809 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:15.753811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:15.753812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:15.753821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:15.755274 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.755737 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.755738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.755739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.755798 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.756046 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.756047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.756047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.756172 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:15.756234 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.756555 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.756651 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 01:05:15.756968 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.745681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43161 (local address 127.0.98.3:41019)
0926 01:05:15.745885 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.745925 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.746494 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.746731 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.746736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.747734 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:15.748036 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.748043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.749219 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.749220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.750725 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.750729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.751373 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.751374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.751521 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.752128 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:15.752139 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:15.753699 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:15.755375 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.755393 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.755396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.755653 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.755860 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.755861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.755861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.755984 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.756233 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:15.756240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:15.756402 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.756516 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.756702 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260926 01:05:15.757455 8376 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43161
I20260926 01:05:15.757546 8376 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:15.757766 8376 heartbeater.cc:511] Master 127.0.98.62:43161 requested a full tablet report, sending...
I20260926 01:05:15.758181 7893 ts_manager.cc:194] Registered new tserver with Master: 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343)
I20260926 01:05:15.758677 7893 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41019
I20260926 01:05:15.760607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 8245
I20260926 01:05:15.760702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:05:15.760879 392 external_mini_cluster.cc:1428] Setting key ee2e37ac250ff5f3e0e73424264bf43c
I20260926 01:05:15.761961 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:05:15.764489 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.758552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43161 (local address 127.0.98.3:57243)
0926 01:05:15.758683 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.758693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.758758 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.758883 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.758883 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.759055 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:15.759115 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.759116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.759787 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.759788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.760308 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.760309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.760650 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.760651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.760714 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.761196 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:15.761202 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:15.761562 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:15.763378 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.763380 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.763381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.763609 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.763851 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:15.763852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:15.763852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:15.763900 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:15.764141 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:15.764142 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:15.764192 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.764252 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.764372 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260926 01:05:15.764483 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.758615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57243 (local address 127.0.98.62:43161)
0926 01:05:15.758659 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.758661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.758764 (+ 103us) server_negotiation.cc:407] Connection header received
0926 01:05:15.758811 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.758812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.758822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.758877 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:15.759195 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.759683 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.760403 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.760587 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.760640 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.761653 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:15.761654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:15.761655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:15.761661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:15.763241 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.763722 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.763723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.763724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.763788 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.763961 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.763961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.763962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.764077 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:15.764137 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.764285 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.764370 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 01:05:15.784456 7943 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:15.778677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.0.98.62:43161)
0926 01:05:15.778754 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.778756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.778765 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:15.778790 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.778790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.778799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.778850 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:15.779566 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.780099 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.780946 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.781137 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.781211 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.784356 (+ 3145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43718: BlockingRecv error: recv got EOF from 127.0.0.1:43718 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 01:05:15 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:05:15.793586 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.785353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.0.98.62:43161)
0926 01:05:15.785535 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.785537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.785548 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:15.785573 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.785574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.785583 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.785643 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:15.786181 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.786751 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.787469 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.787657 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.787721 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.790735 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:15.790737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:15.790737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:15.790748 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:15.792316 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.792767 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.792768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.792768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.792837 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:15.793114 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:15.793115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:15.793115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:15.793245 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:15.793332 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.793445 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.793513 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 01:05:15.795961 7893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:05:15.797706 7893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:05:15.804761 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.800618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36881 (local address 127.0.0.1:43308)
0926 01:05:15.800777 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.800884 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.800957 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.801173 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.801176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.801178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:15.801249 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.801251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.802335 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.802336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.803564 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.803566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.804456 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.804457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.804536 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.804551 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.804618 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.804672 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260926 01:05:15.804804 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.800772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.0.98.1:36881)
0926 01:05:15.801021 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.801026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.801036 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:15.801074 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.801076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.801091 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.801156 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:15.801319 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.802264 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.803662 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.804378 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.804441 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.804520 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.804656 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.804693 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260926 01:05:15.805253 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.801042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46343 (local address 127.0.0.1:43316)
0926 01:05:15.801531 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.801540 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.801581 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.801829 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.801830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.801832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:15.801873 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.801874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.803113 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.803114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.803989 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.803991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.804993 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.804993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.805072 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.805077 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.805106 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.805147 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":58,"threads_started":1}
I20260926 01:05:15.805326 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.801166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.0.98.2:42899)
0926 01:05:15.801392 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.801396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.801405 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:15.801447 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.801449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.801464 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.801533 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:15.801733 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.802860 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.804180 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.804860 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.804921 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.804998 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.805192 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.805237 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260926 01:05:15.805513 8030 tablet_service.cc:1467] Processing CreateTablet for tablet 2dea60796b5a41bba0aab1737e3dd78d (DEFAULT_TABLE table=test-table [id=d281f8012f21432ba45f6b5126ebd81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:15.805660 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.801375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.0.98.3:46343)
0926 01:05:15.801677 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.801680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.801688 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:15.801727 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.801728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.801742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.801815 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:15.801945 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.803023 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.804060 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.804775 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.805450 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.805521 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.805542 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.805574 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":75,"threads_started":1}
I20260926 01:05:15.805883 8030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dea60796b5a41bba0aab1737e3dd78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:15.805275 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.800876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42899 (local address 127.0.0.1:47008)
0926 01:05:15.801153 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.801164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.801215 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.801613 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.801614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.801616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:15.801661 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.801662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.802954 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.802955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.804098 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.804101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.805024 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.805025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.805077 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.805080 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.805121 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.805150 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":7,"thread_start_us":78,"threads_started":1}
I20260926 01:05:15.806697 8310 tablet_service.cc:1467] Processing CreateTablet for tablet 2dea60796b5a41bba0aab1737e3dd78d (DEFAULT_TABLE table=test-table [id=d281f8012f21432ba45f6b5126ebd81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:15.807027 8310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dea60796b5a41bba0aab1737e3dd78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:15.807016 8170 tablet_service.cc:1467] Processing CreateTablet for tablet 2dea60796b5a41bba0aab1737e3dd78d (DEFAULT_TABLE table=test-table [id=d281f8012f21432ba45f6b5126ebd81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:15.807549 8170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dea60796b5a41bba0aab1737e3dd78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:15.809854 8404 tablet_bootstrap.cc:492] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: Bootstrap starting.
I20260926 01:05:15.810552 8405 tablet_bootstrap.cc:492] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: Bootstrap starting.
I20260926 01:05:15.810871 8404 tablet_bootstrap.cc:654] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:15.811159 8404 log.cc:824] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:15.811192 8405 tablet_bootstrap.cc:654] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:15.811509 8405 log.cc:824] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:15.811841 8407 tablet_bootstrap.cc:492] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: Bootstrap starting.
I20260926 01:05:15.812031 8404 tablet_bootstrap.cc:492] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: No bootstrap required, opened a new log
I20260926 01:05:15.812103 8404 ts_tablet_manager.cc:1397] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:05:15.812301 8405 tablet_bootstrap.cc:492] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: No bootstrap required, opened a new log
I20260926 01:05:15.812372 8405 ts_tablet_manager.cc:1397] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:15.812460 8407 tablet_bootstrap.cc:654] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:15.812847 8407 log.cc:824] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:15.813614 8407 tablet_bootstrap.cc:492] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: No bootstrap required, opened a new log
I20260926 01:05:15.813687 8407 ts_tablet_manager.cc:1397] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:15.813853 8404 raft_consensus.cc:348] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.814023 8404 raft_consensus.cc:374] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:15.814057 8404 raft_consensus.cc:729] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927c3fa6a15943e98a45fe6e158dfd89, State: Initialized, Role: FOLLOWER
I20260926 01:05:15.814193 8404 consensus_queue.cc:260] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.814308 8405 raft_consensus.cc:348] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.814487 8404 ts_tablet_manager.cc:1428] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:05:15.814503 8405 raft_consensus.cc:374] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:15.814549 8405 raft_consensus.cc:729] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f2c39fca3144b3903c5ab0d2929d1d, State: Initialized, Role: FOLLOWER
I20260926 01:05:15.814599 8376 heartbeater.cc:503] Master 127.0.98.62:43161 was elected leader, sending a full tablet report...
I20260926 01:05:15.814662 8405 consensus_queue.cc:260] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.814936 8405 ts_tablet_manager.cc:1428] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:05:15.815076 8096 heartbeater.cc:503] Master 127.0.98.62:43161 was elected leader, sending a full tablet report...
I20260926 01:05:15.815666 8407 raft_consensus.cc:348] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.815838 8407 raft_consensus.cc:374] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:15.815881 8407 raft_consensus.cc:729] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a170d255d1084a6a8a5373bd97cd0510, State: Initialized, Role: FOLLOWER
I20260926 01:05:15.815992 8407 consensus_queue.cc:260] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.816255 8407 ts_tablet_manager.cc:1428] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:05:15.816272 8236 heartbeater.cc:503] Master 127.0.98.62:43161 was elected leader, sending a full tablet report...
I20260926 01:05:15.891705 8412 raft_consensus.cc:482] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:15.891841 8412 raft_consensus.cc:504] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.892205 8412 leader_election.cc:288] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881), 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343)
I20260926 01:05:15.895637 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.892633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51189 (local address 127.0.98.3:46343)
0926 01:05:15.892698 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.892699 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.892806 (+ 107us) server_negotiation.cc:407] Connection header received
0926 01:05:15.892847 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.892848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.892861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.892914 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:15.893166 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.893646 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.894776 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.895179 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.895238 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.895248 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.895520 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.895563 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:05:15.895658 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.892423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36881 (local address 127.0.98.2:52931)
0926 01:05:15.892475 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.892488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.892537 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.892769 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.892770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.892772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:15.892846 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.892848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.893673 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.893674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.894683 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.894686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.895378 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.895378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.895455 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.895458 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.895506 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.895552 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":7}
I20260926 01:05:15.895716 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.892566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52931 (local address 127.0.98.1:36881)
0926 01:05:15.892617 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.892619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.892630 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:15.892659 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.892660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.892671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.892763 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:15.892984 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.893543 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.894791 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.895293 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.895369 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.895379 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.895606 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.895650 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:05:15.895891 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.892530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46343 (local address 127.0.98.2:51189)
0926 01:05:15.892731 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.892742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.892801 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.892985 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.892986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.892988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:15.893060 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.893061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.893772 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.893773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.894647 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.894649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.895299 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.895300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.895427 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.895432 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.895504 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.895551 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":118,"threads_started":1}
I20260926 01:05:15.896004 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dea60796b5a41bba0aab1737e3dd78d" candidate_uuid: "a170d255d1084a6a8a5373bd97cd0510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" is_pre_election: true
I20260926 01:05:15.896163 8050 raft_consensus.cc:2454] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a170d255d1084a6a8a5373bd97cd0510 in term 0.
W20260926 01:05:15.896330 8097 tablet.cc:2369] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:05:15.896400 8124 leader_election.cc:302] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f2c39fca3144b3903c5ab0d2929d1d, a170d255d1084a6a8a5373bd97cd0510; no voters:
I20260926 01:05:15.896442 8330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dea60796b5a41bba0aab1737e3dd78d" candidate_uuid: "a170d255d1084a6a8a5373bd97cd0510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927c3fa6a15943e98a45fe6e158dfd89" is_pre_election: true
I20260926 01:05:15.896582 8330 raft_consensus.cc:2454] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a170d255d1084a6a8a5373bd97cd0510 in term 0.
I20260926 01:05:15.896589 8412 raft_consensus.cc:2781] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:05:15.896639 8412 raft_consensus.cc:482] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:15.896668 8412 raft_consensus.cc:3037] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:15.897634 8412 raft_consensus.cc:504] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.897761 8412 leader_election.cc:288] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [CANDIDATE]: Term 1 election: Requested vote from peers 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881), 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343)
I20260926 01:05:15.897948 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dea60796b5a41bba0aab1737e3dd78d" candidate_uuid: "a170d255d1084a6a8a5373bd97cd0510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f2c39fca3144b3903c5ab0d2929d1d"
I20260926 01:05:15.897991 8330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dea60796b5a41bba0aab1737e3dd78d" candidate_uuid: "a170d255d1084a6a8a5373bd97cd0510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927c3fa6a15943e98a45fe6e158dfd89"
I20260926 01:05:15.898033 8050 raft_consensus.cc:3037] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:15.898087 8330 raft_consensus.cc:3037] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:15.898841 8050 raft_consensus.cc:2454] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a170d255d1084a6a8a5373bd97cd0510 in term 1.
I20260926 01:05:15.898904 8330 raft_consensus.cc:2454] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a170d255d1084a6a8a5373bd97cd0510 in term 1.
I20260926 01:05:15.899053 8124 leader_election.cc:302] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 927c3fa6a15943e98a45fe6e158dfd89, a170d255d1084a6a8a5373bd97cd0510; no voters:
I20260926 01:05:15.899214 8412 raft_consensus.cc:2781] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:15.899346 8412 raft_consensus.cc:686] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 1 LEADER]: Becoming Leader. State: Replica: a170d255d1084a6a8a5373bd97cd0510, State: Running, Role: LEADER
I20260926 01:05:15.899437 8412 consensus_queue.cc:237] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.900484 7892 catalog_manager.cc:5495] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 reported cstate change: term changed from 0 to 1, leader changed from <none> to a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a170d255d1084a6a8a5373bd97cd0510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } health_report { overall_health: UNKNOWN } } }
I20260926 01:05:15.923180 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.920920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47022 (local address 127.0.98.2:42899)
0926 01:05:15.920972 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.920974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.920982 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:15.921021 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.921021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.921043 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.921108 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:15.921218 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.921686 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.922521 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.922742 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.922804 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.922929 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:15.923030 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:15.923084 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.923099 (+ 15us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.923118 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:05:15.925107 8330 raft_consensus.cc:1261] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Refusing update from remote peer a170d255d1084a6a8a5373bd97cd0510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:15.925108 8050 raft_consensus.cc:1261] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Refusing update from remote peer a170d255d1084a6a8a5373bd97cd0510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:15.925443 8414 consensus_queue.cc:1035] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:15.925577 8412 consensus_queue.cc:1035] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:15.928732 8418 mvcc.cc:205] Tried to move back new op lower bound from 7333415796425678848 to 7333415796324560896. Current Snapshot: MvccSnapshot[applied={T|T < 7333415796425678848}]
I20260926 01:05:15.930338 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:15.930477 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:15.930795 8419 mvcc.cc:205] Tried to move back new op lower bound from 7333415796425678848 to 7333415796324560896. Current Snapshot: MvccSnapshot[applied={T|T < 7333415796425678848 or (T in {7333415796425678848})}]
I20260926 01:05:15.931375 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609: Sending DeleteTablet for 3 replicas of tablet 2dea60796b5a41bba0aab1737e3dd78d
I20260926 01:05:15.931849 8310 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dea60796b5a41bba0aab1737e3dd78d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43316
I20260926 01:05:15.931828 8030 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dea60796b5a41bba0aab1737e3dd78d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43308
I20260926 01:05:15.932112 8426 tablet_replica.cc:330] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: stopping tablet replica
I20260926 01:05:15.932109 8170 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dea60796b5a41bba0aab1737e3dd78d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47008
I20260926 01:05:15.932255 8426 raft_consensus.cc:2229] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:15.932435 8428 tablet_replica.cc:330] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: stopping tablet replica
I20260926 01:05:15.932529 8426 raft_consensus.cc:2258] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:15.932667 8428 raft_consensus.cc:2229] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:05:15.932621 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:05:15.932808 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:05:15.933007 8427 tablet_replica.cc:330] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: stopping tablet replica
I20260926 01:05:15.933039 8428 raft_consensus.cc:2258] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:15.933116 8427 raft_consensus.cc:2229] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:15.933118 8426 ts_tablet_manager.cc:1905] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:15.933213 8427 raft_consensus.cc:2258] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:15.933606 8427 ts_tablet_manager.cc:1905] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:15.933749 8428 ts_tablet_manager.cc:1905] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:15.934280 8310 tablet_service.cc:1467] Processing CreateTablet for tablet e2999e9b68a24bc786da9f7691bdaa26 (DEFAULT_TABLE table=test-table [id=622640ccce9541859b11ad702e14f6a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:15.934415 8310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2999e9b68a24bc786da9f7691bdaa26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:15.934454 8030 tablet_service.cc:1467] Processing CreateTablet for tablet e2999e9b68a24bc786da9f7691bdaa26 (DEFAULT_TABLE table=test-table [id=622640ccce9541859b11ad702e14f6a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:15.934562 8030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2999e9b68a24bc786da9f7691bdaa26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:15.935664 8404 tablet_bootstrap.cc:492] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: Bootstrap starting.
I20260926 01:05:15.935814 8405 tablet_bootstrap.cc:492] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d: Bootstrap starting.
I20260926 01:05:15.935807 8428 ts_tablet_manager.cc:1918] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:15.935947 8428 log.cc:1196] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/2dea60796b5a41bba0aab1737e3dd78d
I20260926 01:05:15.936055 8404 tablet_bootstrap.cc:654] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:15.936065 8426 ts_tablet_manager.cc:1918] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:15.936129 8426 log.cc:1196] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/2dea60796b5a41bba0aab1737e3dd78d
I20260926 01:05:15.936178 8405 tablet_bootstrap.cc:654] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:15.936398 8426 ts_tablet_manager.cc:1939] T 2dea60796b5a41bba0aab1737e3dd78d P 60f2c39fca3144b3903c5ab0d2929d1d: Deleting consensus metadata
I20260926 01:05:15.936447 8428 ts_tablet_manager.cc:1939] T 2dea60796b5a41bba0aab1737e3dd78d P a170d255d1084a6a8a5373bd97cd0510: Deleting consensus metadata
I20260926 01:05:15.936537 8170 tablet_service.cc:1467] Processing CreateTablet for tablet e2999e9b68a24bc786da9f7691bdaa26 (DEFAULT_TABLE table=test-table [id=622640ccce9541859b11ad702e14f6a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:15.936641 8170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2999e9b68a24bc786da9f7691bdaa26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:15.937067 7880 catalog_manager.cc:4841] TS a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899): tablet 2dea60796b5a41bba0aab1737e3dd78d (table test-table [id=d281f8012f21432ba45f6b5126ebd81b]) successfully deleted
I20260926 01:05:15.937170 8405 tablet_bootstrap.cc:492] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d: No bootstrap required, opened a new log
I20260926 01:05:15.937215 8405 ts_tablet_manager.cc:1397] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:15.937525 8405 raft_consensus.cc:348] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.937593 8405 raft_consensus.cc:374] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:15.937610 8405 raft_consensus.cc:729] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f2c39fca3144b3903c5ab0d2929d1d, State: Initialized, Role: FOLLOWER
I20260926 01:05:15.937661 8405 consensus_queue.cc:260] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.937772 8405 ts_tablet_manager.cc:1428] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:15.937911 8407 tablet_bootstrap.cc:492] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510: Bootstrap starting.
I20260926 01:05:15.938336 8407 tablet_bootstrap.cc:654] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:15.939043 8407 tablet_bootstrap.cc:492] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510: No bootstrap required, opened a new log
I20260926 01:05:15.939096 8407 ts_tablet_manager.cc:1397] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:15.939283 8427 ts_tablet_manager.cc:1918] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:15.939280 8407 raft_consensus.cc:348] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.939373 8407 raft_consensus.cc:374] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:15.939397 8407 raft_consensus.cc:729] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a170d255d1084a6a8a5373bd97cd0510, State: Initialized, Role: FOLLOWER
I20260926 01:05:15.939432 8404 tablet_bootstrap.cc:492] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: No bootstrap required, opened a new log
I20260926 01:05:15.939436 8407 consensus_queue.cc:260] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.939491 8404 ts_tablet_manager.cc:1397] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 01:05:15.939517 8407 ts_tablet_manager.cc:1428] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:15.939545 8427 log.cc:1196] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/2dea60796b5a41bba0aab1737e3dd78d
I20260926 01:05:15.939642 8404 raft_consensus.cc:348] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.939746 8404 raft_consensus.cc:374] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:15.939774 8404 raft_consensus.cc:729] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927c3fa6a15943e98a45fe6e158dfd89, State: Initialized, Role: FOLLOWER
I20260926 01:05:15.939816 8404 consensus_queue.cc:260] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.939918 8404 ts_tablet_manager.cc:1428] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:15.940068 8427 ts_tablet_manager.cc:1939] T 2dea60796b5a41bba0aab1737e3dd78d P 927c3fa6a15943e98a45fe6e158dfd89: Deleting consensus metadata
I20260926 01:05:15.940586 7880 catalog_manager.cc:4841] TS 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343): tablet 2dea60796b5a41bba0aab1737e3dd78d (table test-table [id=d281f8012f21432ba45f6b5126ebd81b]) successfully deleted
I20260926 01:05:15.942963 7880 catalog_manager.cc:4841] TS 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881): tablet 2dea60796b5a41bba0aab1737e3dd78d (table test-table [id=d281f8012f21432ba45f6b5126ebd81b]) successfully deleted
I20260926 01:05:15.991976 8410 raft_consensus.cc:482] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:15.992106 8410 raft_consensus.cc:504] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.992487 8410 leader_election.cc:288] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881), a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899)
I20260926 01:05:15.996070 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.992711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36881 (local address 127.0.98.3:35133)
0926 01:05:15.992794 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.992807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.992875 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.993068 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.993069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.993070 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:15.993139 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.993141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.993840 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.993841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.995035 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.995038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.995825 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.995826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.995887 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.995890 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.995932 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.995972 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":3}
I20260926 01:05:15.996084 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.992996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39739 (local address 127.0.98.2:42899)
0926 01:05:15.993040 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.993041 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.993052 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:15.993080 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.993080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.993089 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.993153 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:15.993336 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.994014 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.995316 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.995694 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.995758 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.995771 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.995970 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.996013 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:05:15.996125 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.992878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35133 (local address 127.0.98.1:36881)
0926 01:05:15.992921 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:15.992923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:15.992930 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:05:15.992959 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:15.992960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:15.992968 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:15.993018 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:15.993220 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.993754 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.995153 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.995738 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.995827 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.995839 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:15.996022 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 01:05:15.996067 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:05:15.996255 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:15.992827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42899 (local address 127.0.98.3:39739)
0926 01:05:15.992979 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:15.992988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:15.993024 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:15.993161 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:15.993162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:15.993163 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:15.993225 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.993226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.994116 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.994117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.995234 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:15.995236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:15.995796 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:15.995798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:15.995869 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:15.995873 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:15.995930 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 01:05:15.995972 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":19,"thread_start_us":80,"threads_started":1}
I20260926 01:05:15.996510 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2999e9b68a24bc786da9f7691bdaa26" candidate_uuid: "927c3fa6a15943e98a45fe6e158dfd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" is_pre_election: true
I20260926 01:05:15.996598 8050 raft_consensus.cc:2454] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927c3fa6a15943e98a45fe6e158dfd89 in term 0.
I20260926 01:05:15.996783 8190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2999e9b68a24bc786da9f7691bdaa26" candidate_uuid: "927c3fa6a15943e98a45fe6e158dfd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a170d255d1084a6a8a5373bd97cd0510" is_pre_election: true
I20260926 01:05:15.996793 8264 leader_election.cc:302] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f2c39fca3144b3903c5ab0d2929d1d, 927c3fa6a15943e98a45fe6e158dfd89; no voters:
I20260926 01:05:15.996914 8190 raft_consensus.cc:2454] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927c3fa6a15943e98a45fe6e158dfd89 in term 0.
I20260926 01:05:15.996965 8410 raft_consensus.cc:2781] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:05:15.997005 8410 raft_consensus.cc:482] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:15.997025 8410 raft_consensus.cc:3037] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:15.997921 8410 raft_consensus.cc:504] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:15.998077 8410 leader_election.cc:288] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [CANDIDATE]: Term 1 election: Requested vote from peers 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881), a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899)
I20260926 01:05:15.998327 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2999e9b68a24bc786da9f7691bdaa26" candidate_uuid: "927c3fa6a15943e98a45fe6e158dfd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f2c39fca3144b3903c5ab0d2929d1d"
I20260926 01:05:15.998451 8050 raft_consensus.cc:3037] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:15.998438 8190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2999e9b68a24bc786da9f7691bdaa26" candidate_uuid: "927c3fa6a15943e98a45fe6e158dfd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a170d255d1084a6a8a5373bd97cd0510"
I20260926 01:05:15.998510 8190 raft_consensus.cc:3037] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:15.999266 8190 raft_consensus.cc:2454] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927c3fa6a15943e98a45fe6e158dfd89 in term 1.
I20260926 01:05:15.999296 8050 raft_consensus.cc:2454] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927c3fa6a15943e98a45fe6e158dfd89 in term 1.
I20260926 01:05:15.999447 8264 leader_election.cc:302] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f2c39fca3144b3903c5ab0d2929d1d, 927c3fa6a15943e98a45fe6e158dfd89; no voters:
I20260926 01:05:15.999532 8410 raft_consensus.cc:2781] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:15.999681 8410 raft_consensus.cc:686] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 LEADER]: Becoming Leader. State: Replica: 927c3fa6a15943e98a45fe6e158dfd89, State: Running, Role: LEADER
I20260926 01:05:15.999759 8410 consensus_queue.cc:237] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.000533 7893 catalog_manager.cc:5495] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "927c3fa6a15943e98a45fe6e158dfd89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } health_report { overall_health: HEALTHY } } }
W20260926 01:05:16.003716 8377 tablet.cc:2369] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 01:05:16 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:05:16.043536 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.035195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43161 (local address 127.0.0.1:43744)
0926 01:05:16.035359 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:16.035372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:16.035451 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:16.035598 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:16.035599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:16.036035 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:16.036096 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.036099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.036813 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.036814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.037605 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.037608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.038023 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.038024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.038086 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.038680 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:16.038685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:16.040464 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:16.042277 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:16.042300 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:16.042303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:16.042570 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:16.042753 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:16.042754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:16.042754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:16.042817 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:16.043152 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:16.043159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:16.043208 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 01:05:16.043277 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 01:05:16.043388 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260926 01:05:16.043588 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.035287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.0.98.62:43161)
0926 01:05:16.035336 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.035338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.035443 (+ 105us) server_negotiation.cc:407] Connection header received
0926 01:05:16.035514 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.035515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.035524 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.035580 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:16.036186 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.036712 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.037722 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.037930 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.037992 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.040575 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:16.040577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:16.040577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:16.040587 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:16.042173 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:16.042635 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:16.042636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:16.042637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:16.042695 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:16.042868 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:16.042868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:16.042869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:16.042999 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:16.043059 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:16.043310 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 01:05:16.043447 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":26}
I20260926 01:05:16.045789 7893 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43744:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 01:05:16.046018 7893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:05:16.047508 8030 tablet_service.cc:1467] Processing CreateTablet for tablet b8da7a2b6e2146408e0e944724fadc5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0f6c8109a75f4c6cb061da977320a33a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:05:16.047518 8170 tablet_service.cc:1467] Processing CreateTablet for tablet b8da7a2b6e2146408e0e944724fadc5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0f6c8109a75f4c6cb061da977320a33a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:05:16.047665 8170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8da7a2b6e2146408e0e944724fadc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:16.047665 8030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8da7a2b6e2146408e0e944724fadc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:16.048834 8405 tablet_bootstrap.cc:492] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: Bootstrap starting.
I20260926 01:05:16.049185 8407 tablet_bootstrap.cc:492] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: Bootstrap starting.
I20260926 01:05:16.049269 8405 tablet_bootstrap.cc:654] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:16.049268 8310 tablet_service.cc:1467] Processing CreateTablet for tablet b8da7a2b6e2146408e0e944724fadc5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0f6c8109a75f4c6cb061da977320a33a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 01:05:16.049401 8310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8da7a2b6e2146408e0e944724fadc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:16.049502 8407 tablet_bootstrap.cc:654] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:16.050189 8405 tablet_bootstrap.cc:492] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: No bootstrap required, opened a new log
I20260926 01:05:16.050238 8405 ts_tablet_manager.cc:1397] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:16.050294 8407 tablet_bootstrap.cc:492] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: No bootstrap required, opened a new log
I20260926 01:05:16.050333 8407 ts_tablet_manager.cc:1397] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:16.050411 8405 raft_consensus.cc:348] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.050483 8405 raft_consensus.cc:374] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:16.050458 8407 raft_consensus.cc:348] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.050501 8405 raft_consensus.cc:729] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f2c39fca3144b3903c5ab0d2929d1d, State: Initialized, Role: FOLLOWER
I20260926 01:05:16.050515 8407 raft_consensus.cc:374] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:16.050534 8407 raft_consensus.cc:729] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a170d255d1084a6a8a5373bd97cd0510, State: Initialized, Role: FOLLOWER
I20260926 01:05:16.050544 8405 consensus_queue.cc:260] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.050567 8407 consensus_queue.cc:260] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.050626 8405 ts_tablet_manager.cc:1428] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:16.050645 8407 ts_tablet_manager.cc:1428] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:16.050800 8411 tablet_replica.cc:439] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } } }
I20260926 01:05:16.050880 8411 tablet_replica.cc:442] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:05:16.051011 8404 tablet_bootstrap.cc:492] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: Bootstrap starting.
I20260926 01:05:16.050997 8412 tablet_replica.cc:439] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } } }
I20260926 01:05:16.051066 8412 tablet_replica.cc:442] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:05:16.051466 8404 tablet_bootstrap.cc:654] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:16.054981 8404 tablet_bootstrap.cc:492] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: No bootstrap required, opened a new log
I20260926 01:05:16.055051 8404 ts_tablet_manager.cc:1397] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 01:05:16.055336 8404 raft_consensus.cc:348] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.055401 8404 raft_consensus.cc:374] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:16.055413 8404 raft_consensus.cc:729] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927c3fa6a15943e98a45fe6e158dfd89, State: Initialized, Role: FOLLOWER
I20260926 01:05:16.055452 8404 consensus_queue.cc:260] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.055541 8404 ts_tablet_manager.cc:1428] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:16.055612 8410 tablet_replica.cc:439] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } } }
I20260926 01:05:16.055680 8410 tablet_replica.cc:442] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 01:05:16.062975 8237 tablet.cc:2369] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:05:16.279112 8412 raft_consensus.cc:482] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:16.279213 8412 raft_consensus.cc:504] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.279352 8412 leader_election.cc:288] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881), 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343)
I20260926 01:05:16.279584 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8da7a2b6e2146408e0e944724fadc5d" candidate_uuid: "a170d255d1084a6a8a5373bd97cd0510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" is_pre_election: true
I20260926 01:05:16.279691 8050 raft_consensus.cc:2454] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a170d255d1084a6a8a5373bd97cd0510 in term 0.
I20260926 01:05:16.279645 8330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8da7a2b6e2146408e0e944724fadc5d" candidate_uuid: "a170d255d1084a6a8a5373bd97cd0510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927c3fa6a15943e98a45fe6e158dfd89" is_pre_election: true
I20260926 01:05:16.279729 8330 raft_consensus.cc:2454] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a170d255d1084a6a8a5373bd97cd0510 in term 0.
I20260926 01:05:16.279873 8124 leader_election.cc:302] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f2c39fca3144b3903c5ab0d2929d1d, a170d255d1084a6a8a5373bd97cd0510; no voters:
I20260926 01:05:16.279987 8412 raft_consensus.cc:2781] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:05:16.280038 8412 raft_consensus.cc:482] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:16.280057 8412 raft_consensus.cc:3037] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:16.280776 8412 raft_consensus.cc:504] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.280900 8412 leader_election.cc:288] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [CANDIDATE]: Term 1 election: Requested vote from peers 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881), 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343)
I20260926 01:05:16.281173 8330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8da7a2b6e2146408e0e944724fadc5d" candidate_uuid: "a170d255d1084a6a8a5373bd97cd0510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927c3fa6a15943e98a45fe6e158dfd89"
I20260926 01:05:16.281199 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8da7a2b6e2146408e0e944724fadc5d" candidate_uuid: "a170d255d1084a6a8a5373bd97cd0510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f2c39fca3144b3903c5ab0d2929d1d"
I20260926 01:05:16.281248 8330 raft_consensus.cc:3037] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:16.281280 8050 raft_consensus.cc:3037] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:16.281850 8330 raft_consensus.cc:2454] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a170d255d1084a6a8a5373bd97cd0510 in term 1.
I20260926 01:05:16.281874 8050 raft_consensus.cc:2454] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a170d255d1084a6a8a5373bd97cd0510 in term 1.
I20260926 01:05:16.282038 8124 leader_election.cc:302] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 927c3fa6a15943e98a45fe6e158dfd89, a170d255d1084a6a8a5373bd97cd0510; no voters:
I20260926 01:05:16.282140 8412 raft_consensus.cc:2781] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:16.282194 8412 raft_consensus.cc:686] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [term 1 LEADER]: Becoming Leader. State: Replica: a170d255d1084a6a8a5373bd97cd0510, State: Running, Role: LEADER
I20260926 01:05:16.282243 8412 consensus_queue.cc:237] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.282537 8414 tablet_replica.cc:439] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: TxnStatusTablet state changed. Reason: New leader a170d255d1084a6a8a5373bd97cd0510. Latest consensus state: current_term: 1 leader_uuid: "a170d255d1084a6a8a5373bd97cd0510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } } }
I20260926 01:05:16.282609 8414 tablet_replica.cc:442] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:05:16.282735 8445 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 01:05:16.282848 8445 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 01:05:16.282905 7893 catalog_manager.cc:5495] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 reported cstate change: term changed from 0 to 1, leader changed from <none> to a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a170d255d1084a6a8a5373bd97cd0510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } health_report { overall_health: UNKNOWN } } }
I20260926 01:05:16.339051 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.336767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.0.98.2:42899)
0926 01:05:16.336826 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.336827 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.336836 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:16.336909 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.336910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.336924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.336978 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:16.337111 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.337577 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.338310 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.338550 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.338609 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.338785 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:16.338822 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:16.338864 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:16.338937 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 01:05:16.338961 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:05:16.339059 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.336688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42899 (local address 127.0.0.1:47036)
0926 01:05:16.336769 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:16.336781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:16.336824 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:16.336992 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:16.336993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:16.336994 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:05:16.337047 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.337050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.337659 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.337660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.338226 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.338228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.338617 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.338618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.338686 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.338704 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:05:16.338886 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:05:16.338893 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 01:05:16.338934 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 01:05:16.338984 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 01:05:16.340201 8330 raft_consensus.cc:1261] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Refusing update from remote peer a170d255d1084a6a8a5373bd97cd0510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:16.340226 8050 raft_consensus.cc:1261] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Refusing update from remote peer a170d255d1084a6a8a5373bd97cd0510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:16.340617 8414 consensus_queue.cc:1035] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:16.340770 8412 consensus_queue.cc:1035] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:16.341409 8411 tablet_replica.cc:439] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: TxnStatusTablet state changed. Reason: New leader a170d255d1084a6a8a5373bd97cd0510. Latest consensus state: current_term: 1 leader_uuid: "a170d255d1084a6a8a5373bd97cd0510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } } }
I20260926 01:05:16.341516 8411 tablet_replica.cc:442] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:05:16.341643 8410 tablet_replica.cc:439] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: TxnStatusTablet state changed. Reason: New leader a170d255d1084a6a8a5373bd97cd0510. Latest consensus state: current_term: 1 leader_uuid: "a170d255d1084a6a8a5373bd97cd0510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } } }
I20260926 01:05:16.341745 8410 tablet_replica.cc:442] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:05:16.342455 8412 tablet_replica.cc:439] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a170d255d1084a6a8a5373bd97cd0510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } } }
I20260926 01:05:16.342546 8412 tablet_replica.cc:442] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:05:16.342917 8414 tablet_replica.cc:439] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a170d255d1084a6a8a5373bd97cd0510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } } }
I20260926 01:05:16.342996 8414 tablet_replica.cc:442] T b8da7a2b6e2146408e0e944724fadc5d P a170d255d1084a6a8a5373bd97cd0510: This TxnStatusTablet replica's current role is: LEADER
I20260926 01:05:16.343453 8411 tablet_replica.cc:439] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a170d255d1084a6a8a5373bd97cd0510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } } }
I20260926 01:05:16.343546 8411 tablet_replica.cc:442] T b8da7a2b6e2146408e0e944724fadc5d P 60f2c39fca3144b3903c5ab0d2929d1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:05:16.344460 8410 tablet_replica.cc:439] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a170d255d1084a6a8a5373bd97cd0510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } } }
I20260926 01:05:16.344551 8410 tablet_replica.cc:442] T b8da7a2b6e2146408e0e944724fadc5d P 927c3fa6a15943e98a45fe6e158dfd89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 01:05:16.346825 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.343994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.0.98.3:46343)
0926 01:05:16.344033 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.344035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.344044 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:16.344109 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.344110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.344128 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.344194 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:16.344374 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.345042 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.345965 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.346166 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.346221 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.346375 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:16.346535 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:16.346602 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:16.346688 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 01:05:16.346711 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:05:16.353015 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.350146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42899 (local address 127.0.98.3:55049)
0926 01:05:16.350305 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:16.350316 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:16.350374 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:16.350542 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:16.350543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:16.350544 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:05:16.350607 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.350609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.351507 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.351508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.352166 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.352168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.352614 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.352615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.352671 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.352688 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:05:16.352867 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:05:16.352868 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 01:05:16.352897 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 01:05:16.352937 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260926 01:05:16.353017 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.350215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55049 (local address 127.0.98.2:42899)
0926 01:05:16.350264 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.350266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.350403 (+ 137us) server_negotiation.cc:407] Connection header received
0926 01:05:16.350460 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.350461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.350474 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.350536 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:16.350672 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.351426 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.352257 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.352528 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.352605 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.352773 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:16.352810 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:16.352868 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:16.352912 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 01:05:16.352936 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:05:16.355501 8050 raft_consensus.cc:1261] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Refusing update from remote peer 927c3fa6a15943e98a45fe6e158dfd89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:16.355520 8190 raft_consensus.cc:1261] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Refusing update from remote peer 927c3fa6a15943e98a45fe6e158dfd89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:16.355800 8410 consensus_queue.cc:1035] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:16.356011 8433 consensus_queue.cc:1035] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:16.358487 8417 mvcc.cc:205] Tried to move back new op lower bound from 7333415798190268416 to 7333415796735377408. Current Snapshot: MvccSnapshot[applied={T|T < 7333415798190268416}]
I20260926 01:05:16.365281 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.362095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57035 (local address 127.0.98.3:46343)
0926 01:05:16.362142 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.362144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.362241 (+ 97us) server_negotiation.cc:407] Connection header received
0926 01:05:16.362286 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.362287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.362299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.362362 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:16.362561 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.363337 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.364357 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.364672 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.364768 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.365003 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:16.365041 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:16.365087 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:16.365193 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 01:05:16.365216 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:05:16.365324 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.362008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46343 (local address 127.0.98.2:57035)
0926 01:05:16.362166 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:16.362179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:16.362238 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:16.362431 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:16.362432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:16.362434 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 01:05:16.362490 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.362492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.363434 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.363435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.364275 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.364277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.364776 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.364777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.364891 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.364909 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 01:05:16.365114 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 01:05:16.365116 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:05:16.365180 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:05:16.365235 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260926 01:05:16.375237 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:16.375386 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:16.376163 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609: Sending DeleteTablet for 3 replicas of tablet e2999e9b68a24bc786da9f7691bdaa26
I20260926 01:05:16.376516 8030 tablet_service.cc:1514] Processing DeleteTablet for tablet e2999e9b68a24bc786da9f7691bdaa26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43308
I20260926 01:05:16.376537 8170 tablet_service.cc:1514] Processing DeleteTablet for tablet e2999e9b68a24bc786da9f7691bdaa26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47008
I20260926 01:05:16.376636 8426 tablet_replica.cc:330] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d: stopping tablet replica
I20260926 01:05:16.376686 8426 raft_consensus.cc:2229] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:16.376725 8426 raft_consensus.cc:2258] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:16.376796 8428 tablet_replica.cc:330] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510: stopping tablet replica
I20260926 01:05:16.376861 8428 raft_consensus.cc:2229] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:16.376904 8428 raft_consensus.cc:2258] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:16.377287 8426 ts_tablet_manager.cc:1905] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:16.379055 8426 ts_tablet_manager.cc:1918] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:05:16.379138 8426 log.cc:1196] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/e2999e9b68a24bc786da9f7691bdaa26
I20260926 01:05:16.379174 8428 ts_tablet_manager.cc:1905] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:16.379381 8426 ts_tablet_manager.cc:1939] T e2999e9b68a24bc786da9f7691bdaa26 P 60f2c39fca3144b3903c5ab0d2929d1d: Deleting consensus metadata
I20260926 01:05:16.379820 7880 catalog_manager.cc:4841] TS 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881): tablet e2999e9b68a24bc786da9f7691bdaa26 (table test-table [id=622640ccce9541859b11ad702e14f6a2]) successfully deleted
I20260926 01:05:16.380952 8428 ts_tablet_manager.cc:1918] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:05:16.381021 8428 log.cc:1196] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/e2999e9b68a24bc786da9f7691bdaa26
I20260926 01:05:16.381254 8428 ts_tablet_manager.cc:1939] T e2999e9b68a24bc786da9f7691bdaa26 P a170d255d1084a6a8a5373bd97cd0510: Deleting consensus metadata
I20260926 01:05:16.382332 7880 catalog_manager.cc:4841] TS a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899): tablet e2999e9b68a24bc786da9f7691bdaa26 (table test-table [id=622640ccce9541859b11ad702e14f6a2]) successfully deleted
I20260926 01:05:16.382623 8310 tablet_service.cc:1514] Processing DeleteTablet for tablet e2999e9b68a24bc786da9f7691bdaa26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43316
I20260926 01:05:16.382730 8427 tablet_replica.cc:330] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: stopping tablet replica
I20260926 01:05:16.382776 8427 raft_consensus.cc:2229] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:05:16.382892 8427 raft_consensus.cc:2258] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:16.383237 8427 ts_tablet_manager.cc:1905] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:16.384981 8427 ts_tablet_manager.cc:1918] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:05:16.385051 8427 log.cc:1196] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/e2999e9b68a24bc786da9f7691bdaa26
I20260926 01:05:16.385282 8427 ts_tablet_manager.cc:1939] T e2999e9b68a24bc786da9f7691bdaa26 P 927c3fa6a15943e98a45fe6e158dfd89: Deleting consensus metadata
I20260926 01:05:16.385814 7880 catalog_manager.cc:4841] TS 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343): tablet e2999e9b68a24bc786da9f7691bdaa26 (table test-table [id=622640ccce9541859b11ad702e14f6a2]) successfully deleted
Sep 26 01:05:16 dist-test-slave-l73v krb5kdc[7848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 01:05:16 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:16 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 01:05:16.412449 7943 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:16.405152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.0.98.62:43161)
0926 01:05:16.405201 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.405204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.405371 (+ 167us) server_negotiation.cc:407] Connection header received
0926 01:05:16.405430 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.405431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.405444 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.405508 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:16.406199 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.407150 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.408328 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.408654 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.408728 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.412376 (+ 3648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43756: BlockingRecv error: recv got EOF from 127.0.0.1:43756 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 01:05:16 dist-test-slave-l73v krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 01:05:16.422178 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.413538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.0.98.62:43161)
0926 01:05:16.413579 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.413581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.413916 (+ 335us) server_negotiation.cc:407] Connection header received
0926 01:05:16.413976 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.413977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.413988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.414036 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:16.414723 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.415524 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.416566 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.416830 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.416901 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.419554 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:16.419555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:16.419556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:16.419564 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:16.421069 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:16.421395 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:16.421396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:16.421396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:16.421458 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:16.421614 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:16.421615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:16.421616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:16.421752 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:16.421847 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:16.421995 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 01:05:16.422076 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:05:16.423982 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:05:16.424173 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:05:16.425503 8030 tablet_service.cc:1467] Processing CreateTablet for tablet 976ae101050944e1990e3924fd4ce763 (DEFAULT_TABLE table=test-table [id=2da7a5f465e347bcae37485b453f4057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:16.425609 8170 tablet_service.cc:1467] Processing CreateTablet for tablet 976ae101050944e1990e3924fd4ce763 (DEFAULT_TABLE table=test-table [id=2da7a5f465e347bcae37485b453f4057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:16.425645 8030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976ae101050944e1990e3924fd4ce763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:16.425706 8170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976ae101050944e1990e3924fd4ce763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:16.425675 8310 tablet_service.cc:1467] Processing CreateTablet for tablet 976ae101050944e1990e3924fd4ce763 (DEFAULT_TABLE table=test-table [id=2da7a5f465e347bcae37485b453f4057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:16.425803 8310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976ae101050944e1990e3924fd4ce763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:16.426970 8407 tablet_bootstrap.cc:492] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510: Bootstrap starting.
I20260926 01:05:16.427134 8404 tablet_bootstrap.cc:492] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89: Bootstrap starting.
I20260926 01:05:16.427273 8407 tablet_bootstrap.cc:654] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:16.427484 8405 tablet_bootstrap.cc:492] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d: Bootstrap starting.
I20260926 01:05:16.427516 8404 tablet_bootstrap.cc:654] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:16.427799 8405 tablet_bootstrap.cc:654] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:16.428138 8404 tablet_bootstrap.cc:492] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89: No bootstrap required, opened a new log
I20260926 01:05:16.428179 8404 ts_tablet_manager.cc:1397] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:16.428318 8404 raft_consensus.cc:348] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.428396 8404 raft_consensus.cc:374] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:16.428416 8404 raft_consensus.cc:729] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927c3fa6a15943e98a45fe6e158dfd89, State: Initialized, Role: FOLLOWER
I20260926 01:05:16.428458 8404 consensus_queue.cc:260] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.428534 8407 tablet_bootstrap.cc:492] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510: No bootstrap required, opened a new log
I20260926 01:05:16.428553 8404 ts_tablet_manager.cc:1428] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:16.428582 8407 ts_tablet_manager.cc:1397] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260926 01:05:16.428689 8407 raft_consensus.cc:348] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.428756 8407 raft_consensus.cc:374] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:16.428776 8407 raft_consensus.cc:729] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a170d255d1084a6a8a5373bd97cd0510, State: Initialized, Role: FOLLOWER
I20260926 01:05:16.428823 8407 consensus_queue.cc:260] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.428898 8407 ts_tablet_manager.cc:1428] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:16.429087 8405 tablet_bootstrap.cc:492] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d: No bootstrap required, opened a new log
I20260926 01:05:16.429126 8405 ts_tablet_manager.cc:1397] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:05:16.429260 8405 raft_consensus.cc:348] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.429332 8405 raft_consensus.cc:374] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:16.429350 8405 raft_consensus.cc:729] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f2c39fca3144b3903c5ab0d2929d1d, State: Initialized, Role: FOLLOWER
I20260926 01:05:16.429390 8405 consensus_queue.cc:260] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.429471 8405 ts_tablet_manager.cc:1428] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:16.627270 8411 raft_consensus.cc:482] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:16.627398 8411 raft_consensus.cc:504] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.627712 8411 leader_election.cc:288] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899), 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343)
I20260926 01:05:16.630754 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.628076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46343 (local address 127.0.98.1:50937)
0926 01:05:16.628192 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:16.628203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:16.628270 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:16.628425 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:16.628426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:16.628428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:16.628487 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.628488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.629108 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.629109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.629914 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.629917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.630483 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.630484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.630548 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.630554 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 01:05:16.630601 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:05:16.630647 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260926 01:05:16.630728 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.628240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50937 (local address 127.0.98.3:46343)
0926 01:05:16.628305 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.628307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.628315 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:16.628340 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.628341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.628349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.628401 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:16.628565 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.629017 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.630024 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.630416 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.630480 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.630493 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:16.630619 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 01:05:16.630651 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 01:05:16.631119 8330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976ae101050944e1990e3924fd4ce763" candidate_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927c3fa6a15943e98a45fe6e158dfd89" is_pre_election: true
I20260926 01:05:16.631204 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.628028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57135 (local address 127.0.98.2:42899)
0926 01:05:16.628094 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.628096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.628106 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:16.628221 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.628223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.628234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.628265 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:16.628482 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.628954 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.630297 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.630706 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.630765 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.630774 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:16.631119 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 01:05:16.631158 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:05:16.631210 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.627911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42899 (local address 127.0.98.1:57135)
0926 01:05:16.628036 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:16.628049 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:16.628107 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:16.628330 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:16.628331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:16.628333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:16.628394 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.628396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.629059 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.629060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.630196 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:16.630198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.630963 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.630965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:16.631038 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.631042 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:16.631098 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:05:16.631137 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260926 01:05:16.631234 8330 raft_consensus.cc:2454] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60f2c39fca3144b3903c5ab0d2929d1d in term 0.
I20260926 01:05:16.631424 7984 leader_election.cc:302] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f2c39fca3144b3903c5ab0d2929d1d, 927c3fa6a15943e98a45fe6e158dfd89; no voters:
I20260926 01:05:16.631592 8411 raft_consensus.cc:2781] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:05:16.631639 8411 raft_consensus.cc:482] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:16.631665 8411 raft_consensus.cc:3037] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:16.631690 8190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976ae101050944e1990e3924fd4ce763" candidate_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a170d255d1084a6a8a5373bd97cd0510" is_pre_election: true
I20260926 01:05:16.631773 8190 raft_consensus.cc:2454] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60f2c39fca3144b3903c5ab0d2929d1d in term 0.
I20260926 01:05:16.632419 8411 raft_consensus.cc:504] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.632540 8411 leader_election.cc:288] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [CANDIDATE]: Term 1 election: Requested vote from peers a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899), 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343)
I20260926 01:05:16.632704 8190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976ae101050944e1990e3924fd4ce763" candidate_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a170d255d1084a6a8a5373bd97cd0510"
I20260926 01:05:16.632784 8190 raft_consensus.cc:3037] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:16.632753 8330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976ae101050944e1990e3924fd4ce763" candidate_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927c3fa6a15943e98a45fe6e158dfd89"
I20260926 01:05:16.632818 8330 raft_consensus.cc:3037] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:16.633546 8330 raft_consensus.cc:2454] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60f2c39fca3144b3903c5ab0d2929d1d in term 1.
I20260926 01:05:16.633546 8190 raft_consensus.cc:2454] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60f2c39fca3144b3903c5ab0d2929d1d in term 1.
I20260926 01:05:16.633728 7984 leader_election.cc:302] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f2c39fca3144b3903c5ab0d2929d1d, a170d255d1084a6a8a5373bd97cd0510; no voters:
I20260926 01:05:16.633852 8411 raft_consensus.cc:2781] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:16.634017 8411 raft_consensus.cc:686] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 LEADER]: Becoming Leader. State: Replica: 60f2c39fca3144b3903c5ab0d2929d1d, State: Running, Role: LEADER
I20260926 01:05:16.634094 8411 consensus_queue.cc:237] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.634912 7892 catalog_manager.cc:5495] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } health_report { overall_health: UNKNOWN } } }
I20260926 01:05:16.659611 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.656634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.0.98.1:36881)
0926 01:05:16.656783 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.656785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.656793 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:16.656880 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.656881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.656901 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.657005 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:16.657232 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.657844 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.658729 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.658969 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.659027 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.659178 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:16.659308 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:16.659362 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:16.659481 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 01:05:16.659500 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260926 01:05:16.660746 8330 raft_consensus.cc:1261] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Refusing update from remote peer 60f2c39fca3144b3903c5ab0d2929d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:16.660779 8190 raft_consensus.cc:1261] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Refusing update from remote peer 60f2c39fca3144b3903c5ab0d2929d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:16.661060 8411 consensus_queue.cc:1035] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:16.661264 8477 consensus_queue.cc:1035] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:16.664736 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:16.664831 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:16.665537 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609: Sending DeleteTablet for 3 replicas of tablet 976ae101050944e1990e3924fd4ce763
I20260926 01:05:16.665833 8030 tablet_service.cc:1514] Processing DeleteTablet for tablet 976ae101050944e1990e3924fd4ce763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43308
I20260926 01:05:16.665930 8426 tablet_replica.cc:330] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d: stopping tablet replica
I20260926 01:05:16.665877 8310 tablet_service.cc:1514] Processing DeleteTablet for tablet 976ae101050944e1990e3924fd4ce763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43316
I20260926 01:05:16.665972 8426 raft_consensus.cc:2229] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:05:16.665993 8427 tablet_replica.cc:330] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89: stopping tablet replica
I20260926 01:05:16.666033 8427 raft_consensus.cc:2229] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:16.666060 8426 raft_consensus.cc:2258] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:16.666075 8427 raft_consensus.cc:2258] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:16.666133 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:05:16.666222 8170 tablet_service.cc:1514] Processing DeleteTablet for tablet 976ae101050944e1990e3924fd4ce763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47008
W20260926 01:05:16.666286 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:05:16.666302 8428 tablet_replica.cc:330] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510: stopping tablet replica
I20260926 01:05:16.666345 8428 raft_consensus.cc:2229] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:16.666426 8428 raft_consensus.cc:2258] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:16.666787 8426 ts_tablet_manager.cc:1905] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:16.667538 8030 tablet_service.cc:1467] Processing CreateTablet for tablet eb793a94336c4371828e52439be1dedf (DEFAULT_TABLE table=test-table [id=ed2145e61a0d4476adfde7fe3efa67a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:16.667646 8428 ts_tablet_manager.cc:1905] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:16.667740 8030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb793a94336c4371828e52439be1dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:16.668624 8310 tablet_service.cc:1467] Processing CreateTablet for tablet eb793a94336c4371828e52439be1dedf (DEFAULT_TABLE table=test-table [id=ed2145e61a0d4476adfde7fe3efa67a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:16.668716 8426 ts_tablet_manager.cc:1918] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:16.668746 8310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb793a94336c4371828e52439be1dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:16.668762 8426 log.cc:1196] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/976ae101050944e1990e3924fd4ce763
I20260926 01:05:16.668960 8426 ts_tablet_manager.cc:1939] T 976ae101050944e1990e3924fd4ce763 P 60f2c39fca3144b3903c5ab0d2929d1d: Deleting consensus metadata
I20260926 01:05:16.669773 8404 tablet_bootstrap.cc:492] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89: Bootstrap starting.
I20260926 01:05:16.669812 8428 ts_tablet_manager.cc:1918] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:16.669880 8428 log.cc:1196] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/976ae101050944e1990e3924fd4ce763
I20260926 01:05:16.670047 8170 tablet_service.cc:1467] Processing CreateTablet for tablet eb793a94336c4371828e52439be1dedf (DEFAULT_TABLE table=test-table [id=ed2145e61a0d4476adfde7fe3efa67a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:16.670078 8428 ts_tablet_manager.cc:1939] T 976ae101050944e1990e3924fd4ce763 P a170d255d1084a6a8a5373bd97cd0510: Deleting consensus metadata
I20260926 01:05:16.670141 8170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb793a94336c4371828e52439be1dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:16.670136 8404 tablet_bootstrap.cc:654] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:16.670773 8404 tablet_bootstrap.cc:492] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89: No bootstrap required, opened a new log
I20260926 01:05:16.670838 8404 ts_tablet_manager.cc:1397] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:16.670979 8404 raft_consensus.cc:348] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.671056 8404 raft_consensus.cc:374] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:16.671072 8404 raft_consensus.cc:729] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927c3fa6a15943e98a45fe6e158dfd89, State: Initialized, Role: FOLLOWER
I20260926 01:05:16.671110 8404 consensus_queue.cc:260] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.671198 8404 ts_tablet_manager.cc:1428] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:16.671382 8407 tablet_bootstrap.cc:492] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510: Bootstrap starting.
I20260926 01:05:16.671563 7880 catalog_manager.cc:4841] TS a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899): tablet 976ae101050944e1990e3924fd4ce763 (table test-table [id=2da7a5f465e347bcae37485b453f4057]) successfully deleted
I20260926 01:05:16.671656 7880 catalog_manager.cc:4841] TS 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881): tablet 976ae101050944e1990e3924fd4ce763 (table test-table [id=2da7a5f465e347bcae37485b453f4057]) successfully deleted
I20260926 01:05:16.671670 8407 tablet_bootstrap.cc:654] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:16.672221 8407 tablet_bootstrap.cc:492] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510: No bootstrap required, opened a new log
I20260926 01:05:16.672273 8407 ts_tablet_manager.cc:1397] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:16.672403 8407 raft_consensus.cc:348] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.672475 8407 raft_consensus.cc:374] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:16.672493 8407 raft_consensus.cc:729] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a170d255d1084a6a8a5373bd97cd0510, State: Initialized, Role: FOLLOWER
I20260926 01:05:16.672533 8407 consensus_queue.cc:260] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.672612 8407 ts_tablet_manager.cc:1428] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:16.672714 8405 tablet_bootstrap.cc:492] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d: Bootstrap starting.
I20260926 01:05:16.673111 8405 tablet_bootstrap.cc:654] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:16.673707 8427 ts_tablet_manager.cc:1905] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:16.673758 8405 tablet_bootstrap.cc:492] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d: No bootstrap required, opened a new log
I20260926 01:05:16.673800 8405 ts_tablet_manager.cc:1397] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:16.673956 8405 raft_consensus.cc:348] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.674026 8405 raft_consensus.cc:374] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:16.674042 8405 raft_consensus.cc:729] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f2c39fca3144b3903c5ab0d2929d1d, State: Initialized, Role: FOLLOWER
I20260926 01:05:16.674080 8405 consensus_queue.cc:260] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.674166 8405 ts_tablet_manager.cc:1428] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:16.675479 8427 ts_tablet_manager.cc:1918] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:16.675547 8427 log.cc:1196] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/976ae101050944e1990e3924fd4ce763
I20260926 01:05:16.675741 8427 ts_tablet_manager.cc:1939] T 976ae101050944e1990e3924fd4ce763 P 927c3fa6a15943e98a45fe6e158dfd89: Deleting consensus metadata
I20260926 01:05:16.676071 7880 catalog_manager.cc:4841] TS 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343): tablet 976ae101050944e1990e3924fd4ce763 (table test-table [id=2da7a5f465e347bcae37485b453f4057]) successfully deleted
I20260926 01:05:16.915803 8410 raft_consensus.cc:482] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:16.915916 8410 raft_consensus.cc:504] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.916074 8410 leader_election.cc:288] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881), a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899)
I20260926 01:05:16.916374 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb793a94336c4371828e52439be1dedf" candidate_uuid: "927c3fa6a15943e98a45fe6e158dfd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" is_pre_election: true
I20260926 01:05:16.916460 8050 raft_consensus.cc:2454] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927c3fa6a15943e98a45fe6e158dfd89 in term 0.
I20260926 01:05:16.916395 8190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb793a94336c4371828e52439be1dedf" candidate_uuid: "927c3fa6a15943e98a45fe6e158dfd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a170d255d1084a6a8a5373bd97cd0510" is_pre_election: true
I20260926 01:05:16.916503 8190 raft_consensus.cc:2454] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927c3fa6a15943e98a45fe6e158dfd89 in term 0.
I20260926 01:05:16.916640 8264 leader_election.cc:302] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f2c39fca3144b3903c5ab0d2929d1d, 927c3fa6a15943e98a45fe6e158dfd89; no voters:
I20260926 01:05:16.916757 8410 raft_consensus.cc:2781] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:05:16.916806 8410 raft_consensus.cc:482] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:16.916826 8410 raft_consensus.cc:3037] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:16.917582 8410 raft_consensus.cc:504] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.917714 8410 leader_election.cc:288] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [CANDIDATE]: Term 1 election: Requested vote from peers 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881), a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899)
I20260926 01:05:16.918135 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb793a94336c4371828e52439be1dedf" candidate_uuid: "927c3fa6a15943e98a45fe6e158dfd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f2c39fca3144b3903c5ab0d2929d1d"
I20260926 01:05:16.918226 8050 raft_consensus.cc:3037] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:16.918231 8190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb793a94336c4371828e52439be1dedf" candidate_uuid: "927c3fa6a15943e98a45fe6e158dfd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a170d255d1084a6a8a5373bd97cd0510"
I20260926 01:05:16.918426 8190 raft_consensus.cc:3037] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:16.918929 8050 raft_consensus.cc:2454] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927c3fa6a15943e98a45fe6e158dfd89 in term 1.
I20260926 01:05:16.919245 8264 leader_election.cc:302] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f2c39fca3144b3903c5ab0d2929d1d, 927c3fa6a15943e98a45fe6e158dfd89; no voters:
I20260926 01:05:16.919322 8190 raft_consensus.cc:2454] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927c3fa6a15943e98a45fe6e158dfd89 in term 1.
I20260926 01:05:16.919529 8410 raft_consensus.cc:2781] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:16.919837 8410 raft_consensus.cc:686] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 LEADER]: Becoming Leader. State: Replica: 927c3fa6a15943e98a45fe6e158dfd89, State: Running, Role: LEADER
I20260926 01:05:16.919943 8410 consensus_queue.cc:237] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } }
I20260926 01:05:16.920783 7892 catalog_manager.cc:5495] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "927c3fa6a15943e98a45fe6e158dfd89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927c3fa6a15943e98a45fe6e158dfd89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46343 } health_report { overall_health: HEALTHY } } }
I20260926 01:05:16.960965 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.958773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.0.98.3:46343)
0926 01:05:16.958820 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:16.958822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:16.958831 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:16.958870 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:16.958871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:16.958879 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:16.958933 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:16.959068 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.959562 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.960336 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:16.960554 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:16.960609 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:16.960766 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:16.960795 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:16.960831 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:16.960871 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 01:05:16.960888 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:05:16.963199 8050 raft_consensus.cc:1261] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Refusing update from remote peer 927c3fa6a15943e98a45fe6e158dfd89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:16.963198 8190 raft_consensus.cc:1261] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Refusing update from remote peer 927c3fa6a15943e98a45fe6e158dfd89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:16.963418 8488 consensus_queue.cc:1035] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a170d255d1084a6a8a5373bd97cd0510" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:16.963539 8410 consensus_queue.cc:1035] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f2c39fca3144b3903c5ab0d2929d1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:16.973615 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:16.973729 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:16.974436 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beac26dde2184e65b6278720aeede609: Sending DeleteTablet for 3 replicas of tablet eb793a94336c4371828e52439be1dedf
I20260926 01:05:16.974704 8030 tablet_service.cc:1514] Processing DeleteTablet for tablet eb793a94336c4371828e52439be1dedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43308
I20260926 01:05:16.974732 8170 tablet_service.cc:1514] Processing DeleteTablet for tablet eb793a94336c4371828e52439be1dedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47008
I20260926 01:05:16.974749 8310 tablet_service.cc:1514] Processing DeleteTablet for tablet eb793a94336c4371828e52439be1dedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43316
I20260926 01:05:16.974830 8428 tablet_replica.cc:330] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510: stopping tablet replica
I20260926 01:05:16.974853 8427 tablet_replica.cc:330] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89: stopping tablet replica
I20260926 01:05:16.974885 8428 raft_consensus.cc:2229] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:16.974916 8428 raft_consensus.cc:2258] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:16.974912 8427 raft_consensus.cc:2229] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:05:16.974917 8426 tablet_replica.cc:330] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d: stopping tablet replica
I20260926 01:05:16.974962 8426 raft_consensus.cc:2229] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:16.974982 8427 raft_consensus.cc:2258] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:16.974999 8426 raft_consensus.cc:2258] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:16.975227 8428 ts_tablet_manager.cc:1905] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:16.975530 8426 ts_tablet_manager.cc:1905] T eb793a94336c4371828e52439be1dedf P 60f2c39fca3144b3903c5ab0d2929d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:16.976805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 7965
I20260926 01:05:16.976845 8427 ts_tablet_manager.cc:1905] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:16.977675 8428 ts_tablet_manager.cc:1918] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:05:16.977746 8428 log.cc:1196] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/eb793a94336c4371828e52439be1dedf
I20260926 01:05:16.977993 8428 ts_tablet_manager.cc:1939] T eb793a94336c4371828e52439be1dedf P a170d255d1084a6a8a5373bd97cd0510: Deleting consensus metadata
I20260926 01:05:16.978343 7880 catalog_manager.cc:4841] TS a170d255d1084a6a8a5373bd97cd0510 (127.0.98.2:42899): tablet eb793a94336c4371828e52439be1dedf (table test-table [id=ed2145e61a0d4476adfde7fe3efa67a1]) successfully deleted
I20260926 01:05:16.978539 8427 ts_tablet_manager.cc:1918] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 01:05:16.978580 8427 log.cc:1196] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/eb793a94336c4371828e52439be1dedf
I20260926 01:05:16.978704 8427 ts_tablet_manager.cc:1939] T eb793a94336c4371828e52439be1dedf P 927c3fa6a15943e98a45fe6e158dfd89: Deleting consensus metadata
I20260926 01:05:16.979348 7880 catalog_manager.cc:4841] TS 927c3fa6a15943e98a45fe6e158dfd89 (127.0.98.3:46343): tablet eb793a94336c4371828e52439be1dedf (table test-table [id=ed2145e61a0d4476adfde7fe3efa67a1]) successfully deleted
W20260926 01:05:16.983240 7880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36881 (error 108)
I20260926 01:05:16.983433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 8105
I20260926 01:05:16.984345 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:16.984150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36881 (local address 127.0.0.1:43326)
0926 01:05:16.984256 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:16.984302 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":45,"threads_started":1}
W20260926 01:05:16.984927 7880 catalog_manager.cc:4584] TS 60f2c39fca3144b3903c5ab0d2929d1d (127.0.98.1:36881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb793a94336c4371828e52439be1dedf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36881: connect: Connection refused (error 111)
I20260926 01:05:16.989069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 8245
I20260926 01:05:16.994309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 7861
2026-09-26T01:05:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3112 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:05:17 dist-test-slave-l73v krb5kdc[8501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:05:17 dist-test-slave-l73v krb5kdc[8501](info): set up 2 sockets
Sep 26 01:05:17 dist-test-slave-l73v krb5kdc[8501](info): commencing operation
krb5kdc: starting...
W20260926 01:05:17.978929 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:05:18 dist-test-slave-l73v krb5kdc[8501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:05:18.115329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46771
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:18.200680 8514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:18.200933 8514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:18.200959 8514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:18.202550 8514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:18.202613 8514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:18.202631 8514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:18.202643 8514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:18.202656 8514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:18.204378 8514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:18.204631 8514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:18.204821 8514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:18.209853 8522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:18.210083 8514 server_base.cc:1034] running on GCE node
W20260926 01:05:18.209853 8520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:18.209853 8519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:18.210492 8514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:18.210842 8514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:18.212062 8514 hybrid_clock.cc:648] HybridClock initialized: now 1790384718212022 us; error 58 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T01:05:18Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1219 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:05:18 dist-test-slave-l73v krb5kdc[8527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:05:18 dist-test-slave-l73v krb5kdc[8527](info): set up 2 sockets
Sep 26 01:05:18 dist-test-slave-l73v krb5kdc[8527](info): commencing operation
krb5kdc: starting...
W20260926 01:05:19.194839 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:05:19 dist-test-slave-l73v krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:05:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:05:19.323220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:19.404397 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:19.404640 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:19.404675 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:19.405994 8540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:19.406047 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:19.406066 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:19.406082 8540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:19.406098 8540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:19.407842 8540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:19.408120 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:19.408346 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:19.413853 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:19.413861 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:19.413889 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:19.414083 8540 server_base.cc:1034] running on GCE node
I20260926 01:05:19.414482 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:19.414770 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:19.415949 8540 hybrid_clock.cc:648] HybridClock initialized: now 1790384719415931 us; error 32 us; skew 500 ppm
Sep 26 01:05:19 dist-test-slave-l73v krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:19.424984 8540 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:05:19.425529 8540 webserver.cc:460] Webserver started at http://127.0.98.62:45893/ using document root <none> and password file <none>
I20260926 01:05:19.425735 8540 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:19.425782 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:19.425889 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:19.427054 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "3b4de517a8b743099462e7278d642a10"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "b0de0d853206e56deaca815c9a2dedad"
server_key_iv: "978abe2300a0c15e35c053afbca634d7"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.427398 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b4de517a8b743099462e7278d642a10"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "b0de0d853206e56deaca815c9a2dedad"
server_key_iv: "978abe2300a0c15e35c053afbca634d7"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.428982 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:19.429838 8555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:19.430042 8540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:19.430130 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "3b4de517a8b743099462e7278d642a10"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "b0de0d853206e56deaca815c9a2dedad"
server_key_iv: "978abe2300a0c15e35c053afbca634d7"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.430218 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:19.458891 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:19.461108 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:19.461244 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:19.464891 8540 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44669
I20260926 01:05:19.464928 8607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44669 every 8 connection(s)
I20260926 01:05:19.465453 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:19.466185 8608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:19.468668 8608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10: Bootstrap starting.
I20260926 01:05:19.468750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 8540
I20260926 01:05:19.468848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:19.469012 392 external_mini_cluster.cc:1428] Setting key 9af427af182ccf47c0e0ab76b007c787
I20260926 01:05:19.469362 8608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:19.469714 8608 log.cc:824] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:19.470496 8608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10: No bootstrap required, opened a new log
I20260926 01:05:19.472681 8608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260926 01:05:19.472908 8608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:19.472946 8608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4de517a8b743099462e7278d642a10, State: Initialized, Role: FOLLOWER
I20260926 01:05:19.473078 8608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260926 01:05:19.473151 8608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:19.473179 8608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:19.473215 8608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:19.473888 8608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260926 01:05:19.473999 8608 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4de517a8b743099462e7278d642a10; no voters:
I20260926 01:05:19.474180 8608 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:19.474231 8613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:19.474491 8608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:19.474713 8613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4de517a8b743099462e7278d642a10, State: Running, Role: LEADER
I20260926 01:05:19.474846 8613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260926 01:05:19.475299 8614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b4de517a8b743099462e7278d642a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } } }
I20260926 01:05:19.475381 8614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:19.475594 8622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:19.475934 8615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4de517a8b743099462e7278d642a10. Latest consensus state: current_term: 1 leader_uuid: "3b4de517a8b743099462e7278d642a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } } }
I20260926 01:05:19.476023 8615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [sys.catalog]: This master's current role is: LEADER
Sep 26 01:05:19 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:19.476369 8622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:19.478034 8622 catalog_manager.cc:1347] Generated new cluster ID: eefb630a73214e41912d9a79c2e706c8
I20260926 01:05:19.478093 8622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:19.480201 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:19.469591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.0.98.62:44669)
0926 01:05:19.469803 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:19.469814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:19.469858 (+ 44us) server_negotiation.cc:407] Connection header received
0926 01:05:19.470364 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:19.470398 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:19.470476 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:19.470675 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:19.471446 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.472400 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.473481 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.473920 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.474039 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:19.476800 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:19.476804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:19.476804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:19.476817 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:19.478728 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:19.479073 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:19.479077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:19.479077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:19.479173 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:19.479355 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:19.479356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:19.479356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:19.479603 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:19.479765 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:19.479869 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 01:05:19.480011 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260926 01:05:19.487092 8622 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:19.487898 8622 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:19.491669 8622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10: Generated new TSK 0
I20260926 01:05:19.491889 8622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:05:19.525437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44669
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:19.608234 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:19.608462 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:19.608487 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:19.609781 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:19.609845 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:19.609886 8634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:19.611532 8634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:19.611797 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:19.612020 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:19.617267 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:19.617236 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:19.617233 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:19.617494 8634 server_base.cc:1034] running on GCE node
I20260926 01:05:19.617728 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:19.617972 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:19.619122 8634 hybrid_clock.cc:648] HybridClock initialized: now 1790384719619098 us; error 39 us; skew 500 ppm
Sep 26 01:05:19 dist-test-slave-l73v krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:19.621913 8634 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:05:19.622464 8634 webserver.cc:460] Webserver started at http://127.0.98.1:40871/ using document root <none> and password file <none>
I20260926 01:05:19.622689 8634 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:19.622737 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:19.622867 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:19.623914 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "98035d72410a42e59ed1ae168c18511d"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "f638297c7d5c47d22c76b61d9ef98ba1"
server_key_iv: "f8c77f59f78b4caccdca9d2ac5b564c9"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.624264 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "98035d72410a42e59ed1ae168c18511d"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "f638297c7d5c47d22c76b61d9ef98ba1"
server_key_iv: "f8c77f59f78b4caccdca9d2ac5b564c9"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.625687 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:05:19.626653 8649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:19.626864 8634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:19.626956 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "98035d72410a42e59ed1ae168c18511d"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "f638297c7d5c47d22c76b61d9ef98ba1"
server_key_iv: "f8c77f59f78b4caccdca9d2ac5b564c9"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.627032 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:19.650028 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:19.652287 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:19.652437 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:19.652668 8634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:19.652978 8634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:19.653012 8634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:19.653039 8634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:19.653056 8634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:19.658564 8634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40229
I20260926 01:05:19.658630 8762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40229 every 8 connection(s)
I20260926 01:05:19.658946 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:05:19.660427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 8634
I20260926 01:05:19.660511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:05:19.660689 392 external_mini_cluster.cc:1428] Setting key dc12035657766df8065c9c37b4d3a18b
Sep 26 01:05:19 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:05:19.669421 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:19.659686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50065 (local address 127.0.98.62:44669)
0926 01:05:19.659773 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:19.659775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:19.659921 (+ 146us) server_negotiation.cc:407] Connection header received
0926 01:05:19.661013 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:19.661014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:19.661025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:19.661090 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:19.661781 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.662505 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.663605 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.663830 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.663891 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:19.666271 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:19.666272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:19.666273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:19.666282 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:19.667919 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:19.668455 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:19.668457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:19.668457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:19.668527 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:19.668788 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:19.668788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:19.668789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:19.668911 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:19.668966 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:19.669238 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 01:05:19.669349 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 01:05:19.669708 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:19.659646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:50065)
0926 01:05:19.659797 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:19.659842 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:19.660841 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:19.661150 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:19.661153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:19.661479 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:19.661686 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:19.661691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.662622 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.662624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:19.663509 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:19.663512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.663913 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.663914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:19.664049 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:19.664565 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:19.664573 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:19.666150 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:19.668108 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:19.668117 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:19.668119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:19.668366 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:19.668599 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:19.668599 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:19.668599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:19.668718 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:19.668986 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:19.668991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:19.669126 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 01:05:19.669216 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 01:05:19.669349 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260926 01:05:19.670436 8763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44669
I20260926 01:05:19.670545 8763 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:19.670747 8763 heartbeater.cc:511] Master 127.0.98.62:44669 requested a full tablet report, sending...
I20260926 01:05:19.671265 8572 ts_manager.cc:194] Registered new tserver with Master: 98035d72410a42e59ed1ae168c18511d (127.0.98.1:40229)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 01:05:19.671923 8572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:05:19.690845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44669
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:19.774959 8768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:19.775192 8768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:19.775218 8768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:19.776607 8768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:19.776674 8768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:19.776706 8768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:19.778574 8768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:19.778847 8768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:19.779078 8768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:05:19.784528 8768 server_base.cc:1034] running on GCE node
W20260926 01:05:19.784512 8774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:19.784507 8773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:19.784511 8776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:19.784987 8768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:19.785240 8768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:19.786450 8768 hybrid_clock.cc:648] HybridClock initialized: now 1790384719786425 us; error 43 us; skew 500 ppm
Sep 26 01:05:19 dist-test-slave-l73v krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:19.789359 8768 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:05:19.789939 8768 webserver.cc:460] Webserver started at http://127.0.98.2:40173/ using document root <none> and password file <none>
I20260926 01:05:19.790164 8768 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:19.790216 8768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:19.790347 8768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:19.791442 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "a66ba41fec13406da96c6ec186a96149"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "9190d75b939bccda3be2f02959ef773b"
server_key_iv: "e00cbfd819c9a69cf4f648fd50a95197"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.791797 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a66ba41fec13406da96c6ec186a96149"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "9190d75b939bccda3be2f02959ef773b"
server_key_iv: "e00cbfd819c9a69cf4f648fd50a95197"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.793107 8768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:19.793941 8783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:19.794162 8768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:19.794260 8768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "a66ba41fec13406da96c6ec186a96149"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "9190d75b939bccda3be2f02959ef773b"
server_key_iv: "e00cbfd819c9a69cf4f648fd50a95197"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.794327 8768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:19.817428 8768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:19.819911 8768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:19.820070 8768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:19.820307 8768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:19.820616 8768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:19.820649 8768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:19.820677 8768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:19.820693 8768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:19.826535 8768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38221
I20260926 01:05:19.826572 8896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38221 every 8 connection(s)
I20260926 01:05:19.826913 8768 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:05:19 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:19.836512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 8768
I20260926 01:05:19.836637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:05:19.836856 392 external_mini_cluster.cc:1428] Setting key bbbafd71b9b1e6f011c8da0373c55d11
I20260926 01:05:19.837313 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:19.827728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36927 (local address 127.0.98.62:44669)
0926 01:05:19.827807 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:19.827809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:19.827982 (+ 173us) server_negotiation.cc:407] Connection header received
0926 01:05:19.828956 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:19.828957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:19.828969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:19.829028 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:19.829641 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.830192 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.831373 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.831571 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.831635 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:19.833986 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:19.833988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:19.833988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:19.833995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:19.835569 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:19.836051 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:19.836053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:19.836053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:19.836112 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:19.836518 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:19.836520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:19.836520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:19.836655 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:19.836735 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:19.837139 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 01:05:19.837225 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 01:05:19.837507 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:19.827655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:36927)
0926 01:05:19.827860 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:19.827899 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:19.828815 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:19.829085 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:19.829088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:19.829354 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:19.829547 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:19.829552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.830304 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.830305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:19.831277 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:19.831280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.831651 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.831651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:19.831755 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:19.832260 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:19.832270 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:19.833861 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:19.835677 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:19.835686 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:19.835688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:19.835962 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:19.836217 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:19.836219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:19.836219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:19.836402 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:19.836742 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:19.836751 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:19.836956 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 01:05:19.837082 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 01:05:19.837272 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260926 01:05:19.838003 8897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44669
I20260926 01:05:19.838104 8897 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:19.838299 8897 heartbeater.cc:511] Master 127.0.98.62:44669 requested a full tablet report, sending...
I20260926 01:05:19.838711 8572 ts_manager.cc:194] Registered new tserver with Master: a66ba41fec13406da96c6ec186a96149 (127.0.98.2:38221)
I20260926 01:05:19.839108 8572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36927
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:05:19.863152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44669
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:19.945209 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:19.945451 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:19.945477 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:19.946866 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:19.946930 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:19.946960 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:19.948668 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:19.948915 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:19.949146 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:19.954420 8910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:19.954578 8902 server_base.cc:1034] running on GCE node
W20260926 01:05:19.954448 8908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:19.954500 8907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:19.954969 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:19.955230 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:19.956390 8902 hybrid_clock.cc:648] HybridClock initialized: now 1790384719956377 us; error 34 us; skew 500 ppm
Sep 26 01:05:19 dist-test-slave-l73v krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:19.959008 8902 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:05:19.959501 8902 webserver.cc:460] Webserver started at http://127.0.98.3:42571/ using document root <none> and password file <none>
I20260926 01:05:19.959709 8902 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:19.959758 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:19.959889 8902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:19.960964 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "dbc726aa470a45da9b1c09b3a86762b8"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "71407c4089817f80caf490cd0c96ae27"
server_key_iv: "47a21ac73d1b80f722d6da6ead0036f6"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.961299 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dbc726aa470a45da9b1c09b3a86762b8"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "71407c4089817f80caf490cd0c96ae27"
server_key_iv: "47a21ac73d1b80f722d6da6ead0036f6"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.962659 8902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:05:19.963364 8917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:19.963526 8902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:19.963608 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "dbc726aa470a45da9b1c09b3a86762b8"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "71407c4089817f80caf490cd0c96ae27"
server_key_iv: "47a21ac73d1b80f722d6da6ead0036f6"
server_key_version: "encryptionkey@0"
I20260926 01:05:19.963675 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:19.976068 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:19.978355 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:19.978726 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:19.978984 8902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:19.979307 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:19.979343 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:19.979364 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:19.979384 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:19.984999 8902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34715
I20260926 01:05:19.985047 9030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34715 every 8 connection(s)
I20260926 01:05:19.985363 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:05:19.988126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 8902
I20260926 01:05:19.988221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:05:19.988366 392 external_mini_cluster.cc:1428] Setting key 5b6a566aa3ab55aae0debae726bc840d
Sep 26 01:05:19 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:19.995605 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:19.986146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55005 (local address 127.0.98.62:44669)
0926 01:05:19.986220 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:19.986221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:19.986441 (+ 220us) server_negotiation.cc:407] Connection header received
0926 01:05:19.987434 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:19.987436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:19.987447 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:19.987505 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:19.988229 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.989000 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.990280 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.990523 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.990585 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:19.992903 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:19.992905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:19.992905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:19.992913 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:19.994297 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:19.994734 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:19.994735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:19.994735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:19.994790 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:19.995039 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:19.995040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:19.995041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:19.995169 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:19.995231 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:19.995464 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 01:05:19.995534 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 01:05:19.995795 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:19.986060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:55005)
0926 01:05:19.986290 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:19.986342 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:19.987297 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:19.987608 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:19.987612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:19.987905 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:19.988107 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:19.988111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.989086 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.989087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:19.990159 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:19.990162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:19.990596 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:19.990597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:19.990716 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:19.991192 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:19.991201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:19.992787 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:19.994412 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:19.994424 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:19.994428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:19.994670 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:19.994842 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:19.994843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:19.994843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:19.994964 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:19.995224 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:19.995228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:19.995354 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 01:05:19.995444 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 01:05:19.995572 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260926 01:05:19.996225 9031 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44669
I20260926 01:05:19.996320 9031 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:19.996511 9031 heartbeater.cc:511] Master 127.0.98.62:44669 requested a full tablet report, sending...
I20260926 01:05:19.996906 8572 ts_manager.cc:194] Registered new tserver with Master: dbc726aa470a45da9b1c09b3a86762b8 (127.0.98.3:34715)
I20260926 01:05:19.997346 8572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55005
I20260926 01:05:20.000187 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 01:05:20.081990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 8634
I20260926 01:05:20.086052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 8768
I20260926 01:05:20.090502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 8902
I20260926 01:05:20.094187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 8540
I20260926 01:05:20.098739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44669
--webserver_interface=127.0.98.62
--webserver_port=45893
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:20.182117 9040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:20.182354 9040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:20.182407 9040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:20.183660 9040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:20.183706 9040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:20.183725 9040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:20.183741 9040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:20.183756 9040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:20.185410 9040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:20.185674 9040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:20.185894 9040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:20.190768 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:20.190768 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:20.190809 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:20.191119 9040 server_base.cc:1034] running on GCE node
I20260926 01:05:20.191380 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:20.191723 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:20.192924 9040 hybrid_clock.cc:648] HybridClock initialized: now 1790384720192895 us; error 43 us; skew 500 ppm
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:20.195560 9040 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:05:20.196048 9040 webserver.cc:460] Webserver started at http://127.0.98.62:45893/ using document root <none> and password file <none>
I20260926 01:05:20.196250 9040 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:20.196316 9040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:20.197777 9040 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:20.198530 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:20.198693 9040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:20.198781 9040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "3b4de517a8b743099462e7278d642a10"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "b0de0d853206e56deaca815c9a2dedad"
server_key_iv: "978abe2300a0c15e35c053afbca634d7"
server_key_version: "encryptionkey@0"
I20260926 01:05:20.199146 9040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:20.206725 9040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:20.209553 9040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:20.209702 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:20.213827 9040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44669
I20260926 01:05:20.213918 9107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44669 every 8 connection(s)
I20260926 01:05:20.214468 9040 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:20.215679 9108 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:05:20.216349 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10: Bootstrap starting.
I20260926 01:05:20.223912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 9040
I20260926 01:05:20.224228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40229
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44669
--webserver_port=40871
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:05:20.224476 9108 log.cc:824] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:20.226732 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:05:20.227030 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10: Bootstrap complete.
I20260926 01:05:20.229218 9108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260926 01:05:20.229462 9108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4de517a8b743099462e7278d642a10, State: Initialized, Role: FOLLOWER
I20260926 01:05:20.229599 9108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260926 01:05:20.229697 9108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:20.229750 9108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:20.229790 9108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:05:20.230947 9108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260926 01:05:20.231050 9108 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4de517a8b743099462e7278d642a10; no voters:
I20260926 01:05:20.231225 9108 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:05:20.231274 9112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:05:20.231407 9112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [term 2 LEADER]: Becoming Leader. State: Replica: 3b4de517a8b743099462e7278d642a10, State: Running, Role: LEADER
I20260926 01:05:20.231518 9108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:20.231577 9112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } }
I20260926 01:05:20.231895 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3b4de517a8b743099462e7278d642a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } } }
I20260926 01:05:20.232086 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:20.232331 9114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4de517a8b743099462e7278d642a10. Latest consensus state: current_term: 2 leader_uuid: "3b4de517a8b743099462e7278d642a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4de517a8b743099462e7278d642a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44669 } } }
I20260926 01:05:20.232474 9114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:20.232601 9118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:20.233150 9118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:20.233316 9118 catalog_manager.cc:1259] Loaded cluster ID: eefb630a73214e41912d9a79c2e706c8
I20260926 01:05:20.233521 9118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:20.234915 9118 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:20.235096 9118 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b4de517a8b743099462e7278d642a10: Loaded TSK: 0
I20260926 01:05:20.235314 9118 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:05:20.317233 9111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:20.317493 9111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:20.317534 9111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:20.319047 9111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:20.319120 9111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:20.319168 9111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:20.321012 9111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40871
--tserver_master_addrs=127.0.98.62:44669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:20.321301 9111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:20.321545 9111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:05:20.327335 9111 server_base.cc:1034] running on GCE node
W20260926 01:05:20.327247 9134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:20.327191 9137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:20.327188 9135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:20.327764 9111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:20.328038 9111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:20.329248 9111 hybrid_clock.cc:648] HybridClock initialized: now 1790384720329227 us; error 34 us; skew 500 ppm
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384720, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:20.332944 9111 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 01:05:20.333712 9111 webserver.cc:460] Webserver started at http://127.0.98.1:40871/ using document root <none> and password file <none>
I20260926 01:05:20.333999 9111 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:20.334071 9111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:20.335716 9111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:20.336577 9144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:20.336798 9111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:20.336902 9111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "98035d72410a42e59ed1ae168c18511d"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "f638297c7d5c47d22c76b61d9ef98ba1"
server_key_iv: "f8c77f59f78b4caccdca9d2ac5b564c9"
server_key_version: "encryptionkey@0"
I20260926 01:05:20.337450 9111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:20.363775 9111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:20.366343 9111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:20.366547 9111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:20.366792 9111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:20.367143 9111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:20.367187 9111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:20.367223 9111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:20.367246 9111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:20.373930 9111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40229
I20260926 01:05:20.374022 9257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40229 every 8 connection(s)
I20260926 01:05:20.374457 9111 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:05:20.380242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 9111
I20260926 01:05:20.380566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38221
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44669
--webserver_port=40173
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:20.383721 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:20.375236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:55347)
0926 01:05:20.375415 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:20.375461 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:20.376015 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:20.376434 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:20.376438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:20.377068 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:20.377261 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.377266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.378425 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.378426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.379306 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.379309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.379903 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.379904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.380059 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.380651 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:20.380661 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:20.383511 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
W20260926 01:05:20.383868 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:20.375510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55347 (local address 127.0.98.62:44669)
0926 01:05:20.375843 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:20.375852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:20.375900 (+ 48us) server_negotiation.cc:407] Connection header received
0926 01:05:20.376220 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:20.376225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:20.376270 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:20.376387 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:20.377401 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.378284 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.379402 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.379814 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.379911 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.383691 (+ 3780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55347: BlockingRecv error: recv got EOF from 127.0.98.1:55347 (error 108)
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":192,"threads_started":1}
W20260926 01:05:20.383951 9258 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44669 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44669: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:20.389726 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:20.384173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:45181)
0926 01:05:20.384221 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:20.384231 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:20.384272 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:20.384427 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:20.384434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:20.384584 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:20.384649 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.384651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.385351 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.385352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.386162 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.386166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.386613 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.386614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.386683 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.387240 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:20.387245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:20.389644 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 01:05:20.389835 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:20.384239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45181 (local address 127.0.98.62:44669)
0926 01:05:20.384283 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:20.384285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:20.384293 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:20.384319 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:20.384319 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:20.384330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:20.384417 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:20.384773 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.385267 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.386270 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.386537 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.386601 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.389770 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45181: BlockingRecv error: recv got EOF from 127.0.98.1:45181 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:20.395133 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:20.390015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:60911)
0926 01:05:20.390169 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:20.390180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:20.390233 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:20.390492 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:20.390493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:20.390645 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:20.390693 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.390695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.391398 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.391399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.392239 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.392241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.392645 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.392646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.392701 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.393114 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:20.393119 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:20.395071 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
W20260926 01:05:20.395272 9258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:05:20.395314 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:20.390117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60911 (local address 127.0.98.62:44669)
0926 01:05:20.390156 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:20.390158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:20.390230 (+ 72us) server_negotiation.cc:407] Connection header received
0926 01:05:20.390357 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:20.390359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:20.390371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:20.390469 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:20.390791 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.391281 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.392375 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.392570 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.392636 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.395243 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60911: BlockingRecv error: recv got EOF from 127.0.98.1:60911 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 01:05:20.468087 9262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:20.468323 9262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:20.468349 9262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:20.469895 9262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:20.469969 9262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:20.470008 9262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:20.472157 9262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40173
--tserver_master_addrs=127.0.98.62:44669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:20.472404 9262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:20.472601 9262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:20.477645 9269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:20.477638 9271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:20.477772 9268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:20.477860 9262 server_base.cc:1034] running on GCE node
I20260926 01:05:20.478132 9262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:20.478411 9262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:20.479600 9262 hybrid_clock.cc:648] HybridClock initialized: now 1790384720479579 us; error 39 us; skew 500 ppm
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384720, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:20.482206 9262 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 01:05:20.482748 9262 webserver.cc:460] Webserver started at http://127.0.98.2:40173/ using document root <none> and password file <none>
I20260926 01:05:20.482987 9262 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:20.483054 9262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:20.484520 9262 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:20.485267 9278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:20.485488 9262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:20.485586 9262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "a66ba41fec13406da96c6ec186a96149"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "9190d75b939bccda3be2f02959ef773b"
server_key_iv: "e00cbfd819c9a69cf4f648fd50a95197"
server_key_version: "encryptionkey@0"
I20260926 01:05:20.485955 9262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:20.509321 9262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:20.511665 9262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:20.511816 9262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:20.512058 9262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:20.512372 9262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:20.512409 9262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:20.512439 9262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:20.512455 9262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:20.518321 9262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38221
I20260926 01:05:20.518450 9391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38221 every 8 connection(s)
I20260926 01:05:20.518764 9262 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:20.526723 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:20.519476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:52499)
0926 01:05:20.519630 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:20.519671 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:20.520159 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:20.520458 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:20.520462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:20.521045 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:20.521245 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.521251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.522038 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.522039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.522924 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.522928 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.523412 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.523414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.523550 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.524224 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:20.524233 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:20.526560 (+ 2327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260926 01:05:20.526716 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:20.519531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52499 (local address 127.0.98.62:44669)
0926 01:05:20.519593 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:20.519595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:20.519747 (+ 152us) server_negotiation.cc:407] Connection header received
0926 01:05:20.520277 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:20.520278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:20.520292 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:20.520393 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:20.521401 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.521907 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.523030 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.523227 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.523300 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.526666 (+ 3366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52499: BlockingRecv error: recv got EOF from 127.0.98.2:52499 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 01:05:20.526979 9392 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44669 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44669: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260926 01:05:20.526981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 9262
I20260926 01:05:20.527262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34715
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44669
--webserver_port=42571
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:20.532191 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:20.527224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:36371)
0926 01:05:20.527272 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:20.527281 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:20.527316 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:20.527522 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:20.527523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:20.527652 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:20.527703 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.527705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.528581 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.528582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.529385 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.529388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.529770 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.529771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.529838 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.530272 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:20.530277 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:20.532117 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 01:05:20.532267 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:20.527283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36371 (local address 127.0.98.62:44669)
0926 01:05:20.527340 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:20.527342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:20.527350 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:20.527418 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:20.527419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:20.527427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:20.527478 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:20.527784 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.528482 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.529493 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.529683 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.529743 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.532215 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36371: BlockingRecv error: recv got EOF from 127.0.98.2:36371 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:20.536900 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:20.532546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:49893)
0926 01:05:20.532601 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:20.532612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:20.532656 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:20.532793 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:20.532793 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:20.532915 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:20.532948 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.532950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.533862 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.533862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.534446 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.534449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.534871 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.534872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.534919 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.535308 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:20.535312 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:20.536844 (+ 1532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 01:05:20.537075 9392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:05:20.537122 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:20.532602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49893 (local address 127.0.98.62:44669)
0926 01:05:20.532665 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:20.532667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:20.532679 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:20.532707 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:20.532708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:20.532717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:20.532788 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:20.533015 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.533768 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.534517 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.534798 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.534878 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.537052 (+ 2174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49893: BlockingRecv error: recv got EOF from 127.0.98.2:49893 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 01:05:20.611411 9395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:20.611675 9395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:20.611718 9395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:20.613039 9395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:20.613099 9395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:20.613140 9395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:20.614859 9395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42571
--tserver_master_addrs=127.0.98.62:44669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:20.615144 9395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:20.615409 9395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:20.620682 9400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:20.620684 9403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:20.620741 9395 server_base.cc:1034] running on GCE node
W20260926 01:05:20.620682 9401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:20.621207 9395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:20.621493 9395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:20.622648 9395 hybrid_clock.cc:648] HybridClock initialized: now 1790384720622594 us; error 70 us; skew 500 ppm
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384720, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:20.625506 9395 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 01:05:20.625989 9395 webserver.cc:460] Webserver started at http://127.0.98.3:42571/ using document root <none> and password file <none>
I20260926 01:05:20.626240 9395 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:20.626315 9395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:20.627843 9395 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:20.628547 9410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:20.628755 9395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:20.628851 9395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "dbc726aa470a45da9b1c09b3a86762b8"
format_stamp: "Formatted at 2026-09-26 01:05:19 on dist-test-slave-l73v"
server_key: "71407c4089817f80caf490cd0c96ae27"
server_key_iv: "47a21ac73d1b80f722d6da6ead0036f6"
server_key_version: "encryptionkey@0"
I20260926 01:05:20.629228 9395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:20.647050 9395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:20.649302 9395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:20.649421 9395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:20.649681 9395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:20.649992 9395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:20.650025 9395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:20.650056 9395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:20.650074 9395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:20.656126 9395 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34715
I20260926 01:05:20.656215 9523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34715 every 8 connection(s)
I20260926 01:05:20.656544 9395 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:20.663180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 9395
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 01:05:20.664436 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:20.657401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45835 (local address 127.0.98.62:44669)
0926 01:05:20.657501 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:20.657504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:20.657685 (+ 181us) server_negotiation.cc:407] Connection header received
0926 01:05:20.658231 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:20.658233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:20.658245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:20.658298 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:20.659303 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.659831 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.660877 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.661070 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.661128 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.664377 (+ 3249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45835: BlockingRecv error: recv got EOF from 127.0.98.3:45835 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 01:05:20.664541 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:20.657339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:45835)
0926 01:05:20.657547 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:20.657595 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:20.658124 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:20.658373 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:20.658405 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:20.658996 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:20.659198 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.659203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.659940 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.659941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.660779 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.660781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.661153 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.661154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.661284 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.661871 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:20.661880 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:20.664275 (+ 2395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
W20260926 01:05:20.664808 9524 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44669 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44669: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:20.669363 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:20.665053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:50723)
0926 01:05:20.665111 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:20.665122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:20.665170 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:20.665281 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:20.665281 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:20.665394 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:20.665459 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.665462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.666045 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.666046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.666742 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.666746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.667138 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.667139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.667202 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.667664 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:20.667669 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:20.669291 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 01:05:20.669438 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:20.665109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50723 (local address 127.0.98.62:44669)
0926 01:05:20.665150 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:20.665152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:20.665160 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:20.665219 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:20.665220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:20.665228 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:20.665271 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:20.665535 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.665972 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.666814 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.667068 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.667120 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.669386 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50723: BlockingRecv error: recv got EOF from 127.0.98.3:50723 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:05:20.671183 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:20.663730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.0.98.62:44669)
0926 01:05:20.663851 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:20.663854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:20.663937 (+ 83us) server_negotiation.cc:407] Connection header received
0926 01:05:20.664013 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:20.664014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:20.664026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:20.664081 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:20.664666 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.665364 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.666211 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.666431 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.666509 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.667906 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:20.667912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:20.667912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:20.667924 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:20.669564 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:20.669908 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:20.669912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:20.669912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:20.670103 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:20.670408 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:20.670409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:20.670410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:20.670684 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:20.670817 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:20.670974 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 01:05:20.671086 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:20 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:20.674491 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:20.669902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:34161)
0926 01:05:20.669952 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:20.669962 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:20.669989 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:20.670155 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:20.670156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:20.670263 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:20.670297 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.670298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.671020 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.671021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.671642 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:20.671645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.672082 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.672084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:20.672143 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.672556 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:20.672561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:20.674401 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 01:05:20.674682 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:20.669960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34161 (local address 127.0.98.62:44669)
0926 01:05:20.669997 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:20.669999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:20.670005 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:05:20.670023 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:20.670023 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:20.670032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:20.670087 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:20.670421 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.670933 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.671722 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:20.671981 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:20.672046 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:20.674622 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34161: BlockingRecv error: recv got EOF from 127.0.98.3:34161 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 01:05:20.674675 9524 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 01:05:21 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:21 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:21.400388 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:21.395661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:42325)
0926 01:05:21.395808 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:21.395821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:21.395876 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:21.396002 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:21.396003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:21.396159 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:21.396207 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:21.396209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.396876 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.396877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:21.397489 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:21.397491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.397917 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.397921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:21.397972 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:21.398485 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:21.398490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:21.400314 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
W20260926 01:05:21.400571 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:21.395769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42325 (local address 127.0.98.62:44669)
0926 01:05:21.395875 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:21.395876 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:21.395885 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:21.395922 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:21.395922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:21.395932 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:21.395978 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:21.396274 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.396783 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.397591 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.397829 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.397885 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:21.400464 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42325: BlockingRecv error: recv got EOF from 127.0.98.1:42325 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
Sep 26 01:05:21 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:21 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:21.542294 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:21.537574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:52775)
0926 01:05:21.537740 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:21.537753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:21.537827 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:21.537953 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:21.537954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:21.538134 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:21.538181 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:21.538183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.538863 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.538865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:21.539414 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:21.539416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.539739 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.539740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:21.539791 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:21.540309 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:21.540314 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:21.542221 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
W20260926 01:05:21.542433 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:21.537686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52775 (local address 127.0.98.62:44669)
0926 01:05:21.537759 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:21.537760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:21.537819 (+ 59us) server_negotiation.cc:407] Connection header received
0926 01:05:21.537884 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:21.537885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:21.537897 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:21.537943 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:21.538257 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.538789 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.539489 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.539679 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.539735 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:21.542329 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52775: BlockingRecv error: recv got EOF from 127.0.98.2:52775 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 01:05:21 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:21 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:21.679702 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:21.675079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:34385)
0926 01:05:21.675233 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:21.675245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:21.675308 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:21.675444 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:21.675445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:21.675609 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:21.675657 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:21.675691 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.676356 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.676358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:21.676905 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:21.676907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.677272 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.677274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:21.677328 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:21.677798 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:21.677803 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:21.679607 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
W20260926 01:05:21.679773 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:21.675210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34385 (local address 127.0.98.62:44669)
0926 01:05:21.675280 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:21.675281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:21.675290 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:21.675379 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:21.675380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:21.675392 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:21.675437 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:21.675769 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.676279 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.676992 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:21.677204 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:21.677262 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:21.679717 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34385: BlockingRecv error: recv got EOF from 127.0.98.3:34385 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 01:05:22 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:22 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:22.406157 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:22.400938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:41381)
0926 01:05:22.401066 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:22.401077 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:22.401137 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:22.401267 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:22.401267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:22.401427 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:22.401475 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:22.401477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.402215 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.402217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:22.402991 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:22.402995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.403536 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.403538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:22.403623 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:22.404231 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:22.404236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:22.406090 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
W20260926 01:05:22.406301 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:22.401025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41381 (local address 127.0.98.62:44669)
0926 01:05:22.401138 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:22.401140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:22.401172 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:05:22.401197 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:22.401197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:22.401208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:22.401256 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:22.401590 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.402111 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.403117 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.403443 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.403510 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:22.406229 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41381: BlockingRecv error: recv got EOF from 127.0.98.1:41381 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
Sep 26 01:05:22 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:22 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:22.547583 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:22.542801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:52163)
0926 01:05:22.542936 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:22.542947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:22.542999 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:22.543247 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:22.543248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:22.543413 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:22.543471 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:22.543473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.544183 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.544184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:22.544748 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:22.544750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.545165 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.545166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:22.545224 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:22.545754 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:22.545759 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:22.547522 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
W20260926 01:05:22.547710 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:22.542874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52163 (local address 127.0.98.62:44669)
0926 01:05:22.542939 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:22.542940 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:22.542989 (+ 49us) server_negotiation.cc:407] Connection header received
0926 01:05:22.543041 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:22.543042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:22.543111 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:22.543184 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:22.543555 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.544076 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.544851 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.545050 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.545125 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:22.547629 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52163: BlockingRecv error: recv got EOF from 127.0.98.2:52163 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 01:05:22 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:22 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:22.685146 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:22.680246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:33463)
0926 01:05:22.680402 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:22.680415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:22.680483 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:22.680621 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:22.680622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:22.680883 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:22.680943 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:22.680946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.681593 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.681595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:22.682133 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:22.682135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.682499 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.682499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:22.682554 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:22.683016 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:22.683021 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:22.685082 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
W20260926 01:05:22.685300 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:22.680395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33463 (local address 127.0.98.62:44669)
0926 01:05:22.680455 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:22.680458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:22.680465 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:05:22.680549 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:22.680550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:22.680563 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:22.680610 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:22.681029 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.681517 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.682217 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:22.682441 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:22.682498 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:22.685236 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33463: BlockingRecv error: recv got EOF from 127.0.98.3:33463 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 01:05:23 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:23 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:23.413066 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:23.406665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:57707)
0926 01:05:23.406813 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:23.406823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:23.406883 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:23.407148 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:23.407149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:23.407586 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:23.407639 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:23.407642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.408895 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.408896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:23.409526 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:23.409530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.409999 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.410000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:23.410051 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:23.410637 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:23.410644 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:23.412987 (+ 2343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
W20260926 01:05:23.413189 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:23.406771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57707 (local address 127.0.98.62:44669)
0926 01:05:23.406932 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:23.406934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:23.406942 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:23.406967 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:23.406968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:23.406978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:23.407059 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:23.407809 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.408743 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.409651 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.409910 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.410029 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:23.413129 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57707: BlockingRecv error: recv got EOF from 127.0.98.1:57707 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
Sep 26 01:05:23 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:23 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:23.553491 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:23.548459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:46631)
0926 01:05:23.548624 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:23.548638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:23.548719 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:23.548850 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:23.548851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:23.549024 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:23.549070 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:23.549071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.550001 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.550003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:23.550607 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:23.550610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.550939 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.550941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:23.550995 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:23.551556 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:23.551562 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:23.553402 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
W20260926 01:05:23.553580 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:23.548497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46631 (local address 127.0.98.62:44669)
0926 01:05:23.548629 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:23.548632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:23.548715 (+ 83us) server_negotiation.cc:407] Connection header received
0926 01:05:23.548789 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:23.548790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:23.548800 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:23.548847 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:23.549216 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.549750 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.550687 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.550877 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.550935 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:23.553520 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46631: BlockingRecv error: recv got EOF from 127.0.98.2:46631 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 01:05:23 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:23 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:23.690301 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:23.685727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:34405)
0926 01:05:23.685855 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:23.685865 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:23.685929 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:23.686076 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:23.686077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:23.686235 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:23.686284 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:23.686286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.687054 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.687055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:23.687574 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:23.687575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.687903 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.687903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:23.687959 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:23.688460 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:23.688466 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:23.690214 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
W20260926 01:05:23.690357 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:23.685811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34405 (local address 127.0.98.62:44669)
0926 01:05:23.685865 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:23.685867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:23.685920 (+ 53us) server_negotiation.cc:407] Connection header received
0926 01:05:23.685992 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:23.685993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:23.686014 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:23.686061 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:23.686411 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.686986 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.687653 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:23.687844 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:23.687901 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:23.690305 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34405: BlockingRecv error: recv got EOF from 127.0.98.3:34405 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 01:05:24 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:24 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:24.418764 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:24.413740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:36821)
0926 01:05:24.413896 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:24.413911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:24.413968 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:24.414131 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:24.414132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:24.414294 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:24.414345 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:24.414347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.415060 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.415061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:24.415613 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:24.415615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.416004 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.416005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:24.416067 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:24.416648 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:24.416653 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:24.418698 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
W20260926 01:05:24.418874 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:24.413861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36821 (local address 127.0.98.62:44669)
0926 01:05:24.413965 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:24.413967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:24.413977 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:24.414018 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:24.414019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:24.414065 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:24.414117 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:24.414442 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.414957 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.415700 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.415897 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.415955 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:24.418802 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36821: BlockingRecv error: recv got EOF from 127.0.98.1:36821 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":45,"threads_started":1}
Sep 26 01:05:24 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:24 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:24.563752 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:24.554088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:44701)
0926 01:05:24.554259 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:24.554270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:24.554339 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:24.554567 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:24.554568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:24.554732 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:24.554781 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:24.554782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.555720 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.555721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:24.556253 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:24.556255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.556615 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.556616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:24.556659 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:24.561668 (+ 5009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:24.561674 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:24.563689 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
W20260926 01:05:24.563923 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:24.554217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44701 (local address 127.0.98.62:44669)
0926 01:05:24.554271 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:24.554273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:24.554345 (+ 72us) server_negotiation.cc:407] Connection header received
0926 01:05:24.554406 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:24.554407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:24.554421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:24.554488 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:24.554903 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.555629 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.556347 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.556547 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.556610 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:24.563855 (+ 7245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44701: BlockingRecv error: recv got EOF from 127.0.98.2:44701 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 01:05:24 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:24 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:24.695173 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:24.690789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:55929)
0926 01:05:24.690943 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:24.690953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:24.691005 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:24.691117 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:24.691117 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:24.691266 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:24.691312 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:24.691313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.691916 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.691917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:24.692448 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:24.692450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.692781 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.692782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:24.692817 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:24.693275 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:24.693280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:24.695109 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":54,"threads_started":1}
W20260926 01:05:24.695286 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:24.690926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55929 (local address 127.0.98.62:44669)
0926 01:05:24.691011 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:24.691012 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:24.691023 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:24.691044 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:24.691045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:24.691055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:24.691101 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:24.691360 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.691854 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.692534 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:24.692721 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:24.692781 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:24.695220 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55929: BlockingRecv error: recv got EOF from 127.0.98.3:55929 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 01:05:25 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:25 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:25.424100 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:25.419364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:45557)
0926 01:05:25.419496 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:25.419506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:25.419556 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:25.419775 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:25.419775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:25.419930 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:25.419979 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:25.419981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.420676 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.420677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:25.421215 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:25.421217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.421572 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.421573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:25.421624 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:25.422149 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:25.422153 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:25.424038 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
W20260926 01:05:25.424207 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:25.419514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45557 (local address 127.0.98.62:44669)
0926 01:05:25.419646 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:25.419648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:25.419657 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:25.419681 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:25.419682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:25.419692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:25.419761 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:25.420058 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.420560 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.421293 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.421477 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.421522 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:25.424138 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45557: BlockingRecv error: recv got EOF from 127.0.98.1:45557 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
Sep 26 01:05:25 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:25 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:25.568856 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:25.564353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:54961)
0926 01:05:25.564524 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:25.564536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:25.564608 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:25.564739 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:25.564739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:25.564894 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:25.564942 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:25.564943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.565619 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.565620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:25.566162 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:25.566165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.566541 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.566542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:25.566599 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:25.567015 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:25.567019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:25.568795 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
W20260926 01:05:25.569001 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:25.564497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54961 (local address 127.0.98.62:44669)
0926 01:05:25.564593 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:25.564594 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:25.564605 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:25.564663 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:25.564664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:25.564676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:25.564724 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:25.564997 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.565536 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.566252 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.566471 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.566526 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:25.568931 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54961: BlockingRecv error: recv got EOF from 127.0.98.2:54961 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 01:05:25 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:25 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:25.700703 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:25.695768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:37173)
0926 01:05:25.695936 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:25.695950 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:25.696025 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:25.696173 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:25.696174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:25.696336 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:25.696384 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:25.696385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.697126 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.697127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:25.697904 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:25.697907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.698285 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.698286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:25.698325 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:25.698807 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:25.698812 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:25.700628 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
W20260926 01:05:25.700838 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:25.695892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37173 (local address 127.0.98.62:44669)
0926 01:05:25.695951 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:25.695952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:25.696013 (+ 61us) server_negotiation.cc:407] Connection header received
0926 01:05:25.696072 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:25.696073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:25.696083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:25.696131 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:25.696487 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.696992 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.697997 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:25.698220 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:25.698282 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:25.700763 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37173: BlockingRecv error: recv got EOF from 127.0.98.3:37173 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 01:05:26 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:26 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:26.429075 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:26.424605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:39241)
0926 01:05:26.424719 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:26.424731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:26.424786 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:26.424917 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:26.424918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:26.425098 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:26.425159 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:26.425161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.425806 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.425807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:26.426373 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:26.426376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.426807 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.426808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:26.426853 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:26.427340 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:26.427346 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:26.429005 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
W20260926 01:05:26.429260 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:26.424673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39241 (local address 127.0.98.62:44669)
0926 01:05:26.424794 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:26.424796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:26.424805 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:26.424830 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:26.424831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:26.424842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:26.424908 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:26.425237 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.425721 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.426507 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.426735 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.426796 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:26.429186 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39241: BlockingRecv error: recv got EOF from 127.0.98.1:39241 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
Sep 26 01:05:26 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:26 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:26.573941 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:26.569294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:57993)
0926 01:05:26.569434 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:26.569445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:26.569501 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:26.569613 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:26.569614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:26.569801 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:26.569848 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:26.569850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.570548 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.570549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:26.571127 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:26.571130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.571480 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.571480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:26.571522 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:26.572041 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:26.572046 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:26.573867 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
W20260926 01:05:26.574023 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:26.569358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57993 (local address 127.0.98.62:44669)
0926 01:05:26.569417 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:26.569420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:26.569494 (+ 74us) server_negotiation.cc:407] Connection header received
0926 01:05:26.569545 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:26.569545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:26.569555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:26.569602 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:26.569901 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.570430 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.571210 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.571412 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.571469 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:26.573963 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57993: BlockingRecv error: recv got EOF from 127.0.98.2:57993 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 01:05:26 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:26 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:26.705637 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:26.701204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:49779)
0926 01:05:26.701354 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:26.701366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:26.701445 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:26.701579 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:26.701580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:26.701751 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:26.701798 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:26.701800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.702512 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.702513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:26.703033 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:26.703035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.703366 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.703367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:26.703421 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:26.703924 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:26.703929 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:26.705557 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
W20260926 01:05:26.705722 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:26.701290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49779 (local address 127.0.98.62:44669)
0926 01:05:26.701342 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:26.701345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:26.701426 (+ 81us) server_negotiation.cc:407] Connection header received
0926 01:05:26.701498 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:26.701498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:26.701511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:26.701561 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:26.701870 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.702431 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.703116 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:26.703304 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:26.703359 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:26.705665 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49779: BlockingRecv error: recv got EOF from 127.0.98.3:49779 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 01:05:27 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:27 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:27.434175 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:27.429644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:59901)
0926 01:05:27.429779 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:27.429789 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:27.429856 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:27.429999 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:27.429999 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:27.430153 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:27.430200 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:27.430202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.430908 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.430909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:27.431430 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:27.431431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.431801 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.431802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:27.431846 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:27.432297 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:27.432301 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:27.434115 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
W20260926 01:05:27.434286 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:27.429774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59901 (local address 127.0.98.62:44669)
0926 01:05:27.429882 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:27.429884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:27.429893 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:27.429917 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:27.429918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:27.429927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:27.429983 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:27.430289 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.430827 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.431497 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.431749 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.431799 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:27.434217 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59901: BlockingRecv error: recv got EOF from 127.0.98.1:59901 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":46,"threads_started":1}
Sep 26 01:05:27 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:27 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:27.579166 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:27.574557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:50185)
0926 01:05:27.574676 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:27.574688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:27.574750 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:27.574943 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:27.574944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:27.575142 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:27.575188 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:27.575190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.575979 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.575980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:27.576534 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:27.576536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.576856 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.576857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:27.576909 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:27.577407 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:27.577413 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:27.579107 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
W20260926 01:05:27.579275 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:27.574643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50185 (local address 127.0.98.62:44669)
0926 01:05:27.574728 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:27.574729 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:27.574741 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:27.574819 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:27.574820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:27.574833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:27.574914 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:27.575305 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.575872 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.576600 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.576782 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.576825 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:27.579209 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50185: BlockingRecv error: recv got EOF from 127.0.98.2:50185 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 01:05:27 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:27 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:27.710999 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:27.706255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:56021)
0926 01:05:27.706453 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:27.706464 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:27.706562 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:27.706723 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:27.706724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:27.706923 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:27.706978 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:27.706980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.707771 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.707772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:27.708313 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:27.708315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.708646 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.708647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:27.708694 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:27.709141 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:27.709147 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:27.710921 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
W20260926 01:05:27.711082 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:27.706431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56021 (local address 127.0.98.62:44669)
0926 01:05:27.706492 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:27.706494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:27.706557 (+ 63us) server_negotiation.cc:407] Connection header received
0926 01:05:27.706631 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:27.706632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:27.706646 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:27.706687 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:27.707082 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.707680 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.708396 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:27.708584 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:27.708636 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:27.711026 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56021: BlockingRecv error: recv got EOF from 127.0.98.3:56021 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 01:05:28 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:28 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:28.439359 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:28.434632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:39917)
0926 01:05:28.434751 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:28.434762 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:28.434806 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:28.435031 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:28.435031 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:28.435190 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:28.435240 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:28.435241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.435982 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.435983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:28.436555 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:28.436557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.436973 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.436974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:28.437028 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:28.437549 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:28.437553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:28.439297 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
W20260926 01:05:28.439538 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:28.434773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39917 (local address 127.0.98.62:44669)
0926 01:05:28.434901 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:28.434903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:28.434910 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:05:28.434938 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:28.434938 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:28.434948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:28.435010 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:28.435330 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.435883 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.436666 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.436872 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.436930 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:28.439459 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39917: BlockingRecv error: recv got EOF from 127.0.98.1:39917 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
Sep 26 01:05:28 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:28 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:28.584141 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:28.579682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:35895)
0926 01:05:28.579825 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:28.579837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:28.579896 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:28.580021 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:28.580021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:28.580192 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:28.580235 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:28.580237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.581042 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.581043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:28.581569 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:28.581570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.581901 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.581902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:28.581956 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:28.582450 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:28.582455 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:28.584063 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
W20260926 01:05:28.584219 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:28.579802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35895 (local address 127.0.98.62:44669)
0926 01:05:28.579867 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:28.579869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:28.579883 (+ 14us) server_negotiation.cc:407] Connection header received
0926 01:05:28.579958 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:28.579959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:28.579970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:28.580016 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:28.580307 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.580956 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.581647 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.581841 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.581895 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:28.584165 (+ 2270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35895: BlockingRecv error: recv got EOF from 127.0.98.2:35895 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 01:05:28 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:28 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:28.716197 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:28.711520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:53545)
0926 01:05:28.711661 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:28.711677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:28.711729 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:28.711901 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:28.711902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:28.712069 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:28.712114 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:28.712115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.712811 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.712812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:28.713340 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:28.713342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.713709 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.713710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:28.713752 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:28.714281 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:28.714286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:28.716115 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
W20260926 01:05:28.716274 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:28.711625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53545 (local address 127.0.98.62:44669)
0926 01:05:28.711705 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:28.711707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:28.711719 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:28.711784 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:28.711785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:28.711796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:28.711852 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:28.712208 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.712713 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.713450 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:28.713638 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:28.713698 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:28.716219 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53545: BlockingRecv error: recv got EOF from 127.0.98.3:53545 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 01:05:29 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:29 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:29.444429 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:29.439881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:52165)
0926 01:05:29.440041 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:29.440055 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:29.440134 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:29.440282 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:29.440282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:29.440437 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:29.440485 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:29.440487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.441166 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.441167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:29.441713 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:29.441715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.442036 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.442037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:29.442074 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:29.442551 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:29.442555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:29.444347 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
W20260926 01:05:29.444480 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:29.439985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52165 (local address 127.0.98.62:44669)
0926 01:05:29.440110 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:29.440112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:29.440122 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:29.440187 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:29.440188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:29.440198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:29.440265 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:29.440549 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.441091 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.441779 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.441968 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.442034 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:29.444416 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52165: BlockingRecv error: recv got EOF from 127.0.98.1:52165 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
Sep 26 01:05:29 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:29 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:29.594977 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:29.584728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:52317)
0926 01:05:29.584840 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:29.584852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:29.584903 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:29.585085 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:29.585085 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:29.585246 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:29.585296 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:29.585298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.585964 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.585966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:29.586577 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:29.586580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.586988 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.586989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:29.587060 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:29.592995 (+ 5935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:29.593001 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:29.594914 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
W20260926 01:05:29.595131 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:29.584837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52317 (local address 127.0.98.62:44669)
0926 01:05:29.584935 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:29.584936 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:29.584947 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:05:29.584977 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:29.584978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:29.584989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:29.585071 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:29.585358 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.585859 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.586660 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.586863 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.586919 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:29.595067 (+ 8148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52317: BlockingRecv error: recv got EOF from 127.0.98.2:52317 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 01:05:29 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:29 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:29.721526 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:29.716879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:55847)
0926 01:05:29.717014 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:29.717026 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:29.717094 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:29.717232 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:29.717233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:29.717399 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:29.717447 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:29.717449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.718099 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.718100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:29.718668 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:29.718671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.719177 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.719178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:29.719237 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:29.719709 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:29.719715 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:29.721446 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
W20260926 01:05:29.721611 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:29.717003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55847 (local address 127.0.98.62:44669)
0926 01:05:29.717083 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:29.717085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:29.717097 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:29.717161 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:29.717162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:29.717176 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:29.717218 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:29.717524 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.718018 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.718806 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:29.719073 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:29.719145 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:29.721555 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55847: BlockingRecv error: recv got EOF from 127.0.98.3:55847 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 01:05:30 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:30 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:30.449509 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:30.444985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:56337)
0926 01:05:30.445154 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:30.445165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:30.445228 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:30.445369 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:30.445370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:30.445579 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:30.445636 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:30.445639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.446321 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.446322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:30.446886 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:30.446888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.447234 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.447235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:30.447271 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:30.447714 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:30.447718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:30.449449 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
W20260926 01:05:30.449647 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:30.445126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56337 (local address 127.0.98.62:44669)
0926 01:05:30.445237 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:30.445239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:30.445251 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:30.445283 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:30.445284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:30.445294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:30.445358 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:30.445717 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.446248 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.446964 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.447178 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.447233 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:30.449565 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56337: BlockingRecv error: recv got EOF from 127.0.98.1:56337 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
Sep 26 01:05:30 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:30 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:30.600484 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:30.595586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:36989)
0926 01:05:30.595748 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:30.595759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:30.595867 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:30.596001 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:30.596002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:30.596166 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:30.596218 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:30.596221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.596886 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.596887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:30.597409 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:30.597410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.597810 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.597810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:30.597848 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:30.598440 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:30.598445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:30.600424 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
W20260926 01:05:30.600612 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:30.595728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36989 (local address 127.0.98.62:44669)
0926 01:05:30.595794 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:30.595797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:30.595859 (+ 62us) server_negotiation.cc:407] Connection header received
0926 01:05:30.595923 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:30.595924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:30.595935 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:30.595985 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:30.596288 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.596821 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.597506 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.597744 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.597809 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:30.600548 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36989: BlockingRecv error: recv got EOF from 127.0.98.2:36989 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 01:05:30 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:30 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:30.727126 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:30.722147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:53921)
0926 01:05:30.722318 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:30.722330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:30.722432 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:30.722650 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:30.722651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:30.722868 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:30.722920 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:30.722923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.723600 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.723601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:30.724124 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:30.724126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.724445 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.724445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:30.724500 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:30.725014 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:30.725019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:30.727064 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
W20260926 01:05:30.727272 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:30.722466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53921 (local address 127.0.98.62:44669)
0926 01:05:30.722530 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:30.722532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:30.722538 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:05:30.722561 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:30.722562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:30.722575 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:30.722622 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:30.723031 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.723522 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.724205 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:30.724385 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:30.724441 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:30.727205 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53921: BlockingRecv error: recv got EOF from 127.0.98.3:53921 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 01:05:31 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:31 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:31.454339 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:31.450111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:33513)
0926 01:05:31.450262 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:31.450272 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:31.450322 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:31.450492 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:31.450493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:31.450647 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:31.450696 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:31.450697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.451305 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.451306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:31.451835 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:31.451837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.452132 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.452132 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:31.452173 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:31.452633 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:31.452638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:31.454277 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
W20260926 01:05:31.454468 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:31.450199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33513 (local address 127.0.98.62:44669)
0926 01:05:31.450329 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:31.450331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:31.450343 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:31.450370 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:31.450371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:31.450412 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:31.450479 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:31.450753 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.451240 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.451897 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.452079 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.452123 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:31.454362 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33513: BlockingRecv error: recv got EOF from 127.0.98.1:33513 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
Sep 26 01:05:31 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:31 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:31.605599 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:31.601020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:40549)
0926 01:05:31.601191 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:31.601203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:31.601300 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:31.601499 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:31.601500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:31.601651 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:31.601698 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:31.601700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.602477 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.602478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:31.603021 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:31.603023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.603340 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.603340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:31.603380 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:31.603821 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:31.603827 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:31.605542 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
W20260926 01:05:31.605695 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:31.601107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40549 (local address 127.0.98.62:44669)
0926 01:05:31.601178 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:31.601181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:31.601283 (+ 102us) server_negotiation.cc:407] Connection header received
0926 01:05:31.601372 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:31.601373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:31.601399 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:31.601463 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:31.601782 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.602350 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.603090 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.603280 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.603333 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:31.605626 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40549: BlockingRecv error: recv got EOF from 127.0.98.2:40549 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 01:05:31 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:31 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:31.732755 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:31.727789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:51655)
0926 01:05:31.727965 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:31.727976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:31.728036 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:31.728210 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:31.728211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:31.728376 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:31.728428 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:31.728429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.729145 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.729146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:31.729674 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:31.729676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.730057 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.730058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:31.730101 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:31.730641 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:31.730645 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:31.732616 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
W20260926 01:05:31.732895 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:31.727927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51655 (local address 127.0.98.62:44669)
0926 01:05:31.727977 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:31.727979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:31.728026 (+ 47us) server_negotiation.cc:407] Connection header received
0926 01:05:31.728087 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:31.728088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:31.728098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:31.728160 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:31.728540 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.729064 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.729767 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:31.729986 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:31.730047 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:31.732823 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51655: BlockingRecv error: recv got EOF from 127.0.98.3:51655 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 01:05:32 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:32 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:32.459496 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:32.454813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:38895)
0926 01:05:32.454937 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:32.454947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:32.455000 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:32.455182 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:32.455183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:32.455363 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:32.455412 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:32.455414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.456086 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.456087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:32.456638 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:32.456639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.456946 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.456946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:32.456986 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:32.457438 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:32.457443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:32.459413 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
W20260926 01:05:32.459613 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:32.454888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38895 (local address 127.0.98.62:44669)
0926 01:05:32.454977 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:32.454979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:32.454989 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:32.455062 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:32.455063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:32.455073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:32.455144 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:32.455481 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.455976 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.456708 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.456894 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.456936 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:32.459538 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38895: BlockingRecv error: recv got EOF from 127.0.98.1:38895 (error 108)
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":43,"threads_started":1}
Sep 26 01:05:32 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:32 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:32.610972 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:32.606106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:44867)
0926 01:05:32.606253 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:32.606264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:32.606329 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:32.606498 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:32.606499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:32.606661 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:32.606709 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:32.606711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.607377 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.607378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:32.607913 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:32.607915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.608300 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.608301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:32.608362 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:32.608881 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:32.608886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:32.610827 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
W20260926 01:05:32.610994 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:32.606237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44867 (local address 127.0.98.62:44669)
0926 01:05:32.606293 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:32.606295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:32.606311 (+ 16us) server_negotiation.cc:407] Connection header received
0926 01:05:32.606422 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:32.606423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:32.606434 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:32.606484 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:32.606764 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.607269 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.608009 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.608211 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.608272 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:32.610939 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44867: BlockingRecv error: recv got EOF from 127.0.98.2:44867 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 01:05:32 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:32 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:32.738173 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:32.733240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:50011)
0926 01:05:32.733401 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:32.733414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:32.733470 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:32.733702 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:32.733704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:32.733863 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:32.733913 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:32.733915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.734670 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.734671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:32.735247 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:32.735249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.735618 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.735619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:32.735681 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:32.736160 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:32.736165 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:32.738062 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
W20260926 01:05:32.738276 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:32.733359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50011 (local address 127.0.98.62:44669)
0926 01:05:32.733449 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:32.733451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:32.733470 (+ 19us) server_negotiation.cc:407] Connection header received
0926 01:05:32.733577 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:32.733578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:32.733591 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:32.733654 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:32.734016 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.734563 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.735335 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:32.735526 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:32.735585 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:32.738217 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50011: BlockingRecv error: recv got EOF from 127.0.98.3:50011 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 01:05:33 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:33 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:33.465032 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:33.460081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:45457)
0926 01:05:33.460331 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:33.460343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:33.460413 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:33.460595 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:33.460596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:33.460773 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:33.460822 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:33.460824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.461586 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.461588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:33.462185 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:33.462187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.462687 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.462688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:33.462759 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:33.463230 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:33.463235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:33.464970 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":71,"threads_started":1}
W20260926 01:05:33.465157 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:33.460223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45457 (local address 127.0.98.62:44669)
0926 01:05:33.460366 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:33.460368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:33.460394 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:05:33.460468 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:33.460469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:33.460479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:33.460541 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:33.460928 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.461483 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.462297 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.462538 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.462618 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:33.465072 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45457: BlockingRecv error: recv got EOF from 127.0.98.1:45457 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
Sep 26 01:05:33 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:33 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:33.616276 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:33.611559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:59833)
0926 01:05:33.611721 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:33.611735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:33.611795 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:33.611970 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:33.611971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:33.612129 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:33.612215 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:33.612216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.612909 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.612910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:33.613539 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:33.613543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.613921 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.613922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:33.613977 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:33.614471 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:33.614476 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:33.616214 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
W20260926 01:05:33.616397 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:33.611790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59833 (local address 127.0.98.62:44669)
0926 01:05:33.611855 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:33.611857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:33.611866 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:33.611889 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:33.611889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:33.611900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:33.611961 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:33.612319 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.612816 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.613637 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.613852 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.613912 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:33.616327 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59833: BlockingRecv error: recv got EOF from 127.0.98.2:59833 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 01:05:33 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:33 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:33.743623 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:33.738838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:58083)
0926 01:05:33.739003 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:33.739027 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:33.739115 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:33.739281 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:33.739282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:33.739457 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:33.739507 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:33.739509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.740211 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.740212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:33.740768 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:33.740771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.741117 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.741119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:33.741174 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:33.741720 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:33.741726 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:33.743562 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
W20260926 01:05:33.743779 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:33.738954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58083 (local address 127.0.98.62:44669)
0926 01:05:33.739017 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:33.739020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:33.739102 (+ 82us) server_negotiation.cc:407] Connection header received
0926 01:05:33.739188 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:33.739189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:33.739205 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:33.739262 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:33.739594 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.740116 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.740854 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:33.741045 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:33.741110 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:33.743697 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58083: BlockingRecv error: recv got EOF from 127.0.98.3:58083 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 01:05:34 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:34 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:34.470108 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:34.465669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:38063)
0926 01:05:34.465834 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:34.465854 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:34.465924 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:34.466083 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:34.466083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:34.466281 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:34.466343 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:34.466346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.467072 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.467073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:34.467605 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:34.467606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.467952 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.467953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:34.467991 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:34.468414 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:34.468419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:34.470046 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
W20260926 01:05:34.470203 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:34.465780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38063 (local address 127.0.98.62:44669)
0926 01:05:34.465902 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:34.465903 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:34.465915 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:05:34.465985 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:34.465986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:34.465998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:34.466061 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:34.466459 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.466993 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.467690 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.467892 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.467947 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:34.470135 (+ 2188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38063: BlockingRecv error: recv got EOF from 127.0.98.1:38063 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
Sep 26 01:05:34 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:34 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:34.626778 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:34.616747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:60775)
0926 01:05:34.616897 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:34.616909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:34.616978 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:34.617115 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:34.617116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:34.617286 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:34.617334 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:34.617336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.617979 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.617980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:34.618578 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:34.618581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.619044 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.619045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:34.619103 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:34.625015 (+ 5912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:34.625021 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:34.626716 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
W20260926 01:05:34.626993 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:34.616874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60775 (local address 127.0.98.62:44669)
0926 01:05:34.616925 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:34.616928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:34.616960 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:05:34.617044 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:34.617045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:34.617056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:34.617096 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:34.617413 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.617902 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.618685 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.618948 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.619031 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:34.626941 (+ 7910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60775: BlockingRecv error: recv got EOF from 127.0.98.2:60775 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 01:05:34 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:34 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:34.748797 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:34.744111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.3:55417)
0926 01:05:34.744288 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:34.744303 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:34.744383 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:34.744554 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:34.744554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:34.744767 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:34.744819 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:34.744822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.745508 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.745510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:34.746042 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:34.746044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.746420 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.746421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:34.746477 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:34.746956 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:34.746960 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:34.748730 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
W20260926 01:05:34.748947 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:34.744272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55417 (local address 127.0.98.62:44669)
0926 01:05:34.744344 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:34.744346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:34.744362 (+ 16us) server_negotiation.cc:407] Connection header received
0926 01:05:34.744457 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:34.744459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:34.744479 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:34.744539 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:34.744909 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.745419 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.746132 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:34.746325 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:34.746415 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:34.748874 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55417: BlockingRecv error: recv got EOF from 127.0.98.3:55417 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 01:05:35 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:35 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:35.475651 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:35.470795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.1:38771)
0926 01:05:35.470986 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:35.471000 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:35.471071 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:35.471207 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:35.471208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:35.471362 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:35.471423 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:35.471424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:35.472066 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:35.472067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:35.472654 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:35.472657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:35.473075 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:35.473076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:35.473130 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:35.473549 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:35.473553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:35.475576 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
W20260926 01:05:35.475750 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:35.470891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38771 (local address 127.0.98.62:44669)
0926 01:05:35.471018 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:35.471019 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:35.471055 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:05:35.471129 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:35.471129 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:35.471140 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:35.471189 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:35.471486 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:35.471997 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:35.472736 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:35.473008 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:35.473062 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:35.475682 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38771: BlockingRecv error: recv got EOF from 127.0.98.1:38771 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
Sep 26 01:05:35 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 01:05:35 dist-test-slave-l73v krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 01:05:35.635783 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:35.627341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44669 (local address 127.0.98.2:51011)
0926 01:05:35.627472 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:35.627483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:35.627553 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:35.627682 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:35.627683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:35.627864 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:35.627918 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:35.627920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:35.628633 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:35.628634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:35.629174 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:35.629176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:35.629548 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:35.629549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:35.629599 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:35.633790 (+ 4191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:35.633795 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:35.635680 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44669: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
W20260926 01:05:35.635856 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:35.627476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51011 (local address 127.0.98.62:44669)
0926 01:05:35.627534 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:35.627536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:35.627545 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:35.627608 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:35.627609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:35.627619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:35.627664 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:35.628015 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:35.628546 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:35.629266 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:35.629468 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:35.629524 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:35.635801 (+ 6277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51011: BlockingRecv error: recv got EOF from 127.0.98.2:51011 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 01:05:35.666936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 9111
I20260926 01:05:35.671077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 9262
I20260926 01:05:35.675038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 9395
I20260926 01:05:35.678758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 9040
2026-09-26T01:05:35Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17463 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T01:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:35Z Disabled control of system clock
I20260926 01:05:35.703872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42283
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 01:05:35.796629 9591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:35.796936 9591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:35.796974 9591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:35.798894 9591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:35.798970 9591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:35.798991 9591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:35.799010 9591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:35.799059 9591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:35.801344 9591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:35.801663 9591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:35.801982 9591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:35.808975 9599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:35.808974 9596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:35.809394 9591 server_base.cc:1034] running on GCE node
W20260926 01:05:35.809506 9597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:35.809767 9591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:35.810102 9591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:35.811276 9591 hybrid_clock.cc:648] HybridClock initialized: now 1790384735811256 us; error 45 us; skew 500 ppm
I20260926 01:05:35.812618 9591 webserver.cc:460] Webserver started at http://127.0.98.62:38101/ using document root <none> and password file <none>
I20260926 01:05:35.812851 9591 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:35.812947 9591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:35.813073 9591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:35.814182 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "ee39c1baf6ac4427903f6119e2029806"
format_stamp: "Formatted at 2026-09-26 01:05:35 on dist-test-slave-l73v"
server_key: "a22846c56aeac949140d2b36d9660c6e"
server_key_iv: "6dcb829fe84c9e85a4be52a34dbae10b"
server_key_version: "encryptionkey@0"
I20260926 01:05:35.814600 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee39c1baf6ac4427903f6119e2029806"
format_stamp: "Formatted at 2026-09-26 01:05:35 on dist-test-slave-l73v"
server_key: "a22846c56aeac949140d2b36d9660c6e"
server_key_iv: "6dcb829fe84c9e85a4be52a34dbae10b"
server_key_version: "encryptionkey@0"
I20260926 01:05:35.815925 9591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:35.816663 9605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:35.816820 9591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:05:35.816918 9591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "ee39c1baf6ac4427903f6119e2029806"
format_stamp: "Formatted at 2026-09-26 01:05:35 on dist-test-slave-l73v"
server_key: "a22846c56aeac949140d2b36d9660c6e"
server_key_iv: "6dcb829fe84c9e85a4be52a34dbae10b"
server_key_version: "encryptionkey@0"
I20260926 01:05:35.816982 9591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:35.835194 9591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:35.835585 9591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:35.835702 9591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:35.841128 9591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42283
I20260926 01:05:35.841207 9657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42283 every 8 connection(s)
I20260926 01:05:35.841517 9591 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:35.842213 9658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:35.844990 9658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806: Bootstrap starting.
I20260926 01:05:35.845595 9658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:35.845856 9658 log.cc:824] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:35.846712 9658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806: No bootstrap required, opened a new log
I20260926 01:05:35.848548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 9591
I20260926 01:05:35.848774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:35.848649 9658 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee39c1baf6ac4427903f6119e2029806" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42283 } }
I20260926 01:05:35.848829 9658 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:35.848872 9658 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee39c1baf6ac4427903f6119e2029806, State: Initialized, Role: FOLLOWER
I20260926 01:05:35.849027 9658 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee39c1baf6ac4427903f6119e2029806" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42283 } }
I20260926 01:05:35.849112 9658 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:35.849148 9658 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:35.849161 392 external_mini_cluster.cc:1428] Setting key 88026cef40c0e3633e27011cf34c2644
I20260926 01:05:35.849197 9658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:35.849963 9658 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee39c1baf6ac4427903f6119e2029806" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42283 } }
I20260926 01:05:35.850082 9658 leader_election.cc:302] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee39c1baf6ac4427903f6119e2029806; no voters:
I20260926 01:05:35.850224 9658 leader_election.cc:288] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:35.850286 9663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:35.850507 9663 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [term 1 LEADER]: Becoming Leader. State: Replica: ee39c1baf6ac4427903f6119e2029806, State: Running, Role: LEADER
I20260926 01:05:35.850538 9658 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:35.850661 9663 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee39c1baf6ac4427903f6119e2029806" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42283 } }
I20260926 01:05:35.851084 9664 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee39c1baf6ac4427903f6119e2029806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee39c1baf6ac4427903f6119e2029806" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42283 } } }
I20260926 01:05:35.851089 9665 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee39c1baf6ac4427903f6119e2029806. Latest consensus state: current_term: 1 leader_uuid: "ee39c1baf6ac4427903f6119e2029806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee39c1baf6ac4427903f6119e2029806" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42283 } } }
I20260926 01:05:35.851189 9664 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:35.851192 9665 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:35.851887 9672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:35.852473 9672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:35.854307 9672 catalog_manager.cc:1347] Generated new cluster ID: 35644340814c4ce991f1a9ae12d9d9a9
I20260926 01:05:35.854476 9672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:35.855888 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:35.849752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.0.98.62:42283)
0926 01:05:35.850011 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:35.850020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:35.850058 (+ 38us) server_negotiation.cc:407] Connection header received
0926 01:05:35.850192 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:35.850200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:35.850281 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:35.850444 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:35.850800 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:35.852497 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:35.853692 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:35.854266 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:35.854422 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:35.855175 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:35.855179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:35.855179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:35.855380 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:35.855421 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:35.855437 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:35.855499 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:35.855579 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 01:05:35.855663 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260926 01:05:35.862401 9672 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:35.863217 9672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:35.874989 9672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee39c1baf6ac4427903f6119e2029806: Generated new TSK 0
I20260926 01:05:35.875227 9672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:05:35.881814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42283
--builtin_ntp_servers=127.0.98.20:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 01:05:35.966338 9682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:35.966595 9682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:35.966619 9682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:35.967928 9682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:35.967983 9682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:35.968011 9682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:35.969439 9682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:35.969707 9682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:35.970021 9682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:35.975052 9690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:35.975198 9682 server_base.cc:1034] running on GCE node
W20260926 01:05:35.975024 9687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:35.975040 9688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:35.975590 9682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:35.975853 9682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:35.977038 9682 hybrid_clock.cc:648] HybridClock initialized: now 1790384735977018 us; error 38 us; skew 500 ppm
I20260926 01:05:35.978482 9682 webserver.cc:460] Webserver started at http://127.0.98.1:33251/ using document root <none> and password file <none>
I20260926 01:05:35.978729 9682 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:35.978780 9682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:35.978911 9682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:35.980018 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "999f56d92b42404ab9cccaf2420c5b73"
format_stamp: "Formatted at 2026-09-26 01:05:35 on dist-test-slave-l73v"
server_key: "f85fe5dfbad73706736f0850d49be71c"
server_key_iv: "ee35506c3abc3c3c1d9b6e5aa799a69d"
server_key_version: "encryptionkey@0"
I20260926 01:05:35.980367 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "999f56d92b42404ab9cccaf2420c5b73"
format_stamp: "Formatted at 2026-09-26 01:05:35 on dist-test-slave-l73v"
server_key: "f85fe5dfbad73706736f0850d49be71c"
server_key_iv: "ee35506c3abc3c3c1d9b6e5aa799a69d"
server_key_version: "encryptionkey@0"
I20260926 01:05:35.981639 9682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:05:35.982455 9696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:35.982625 9682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:35.982717 9682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "999f56d92b42404ab9cccaf2420c5b73"
format_stamp: "Formatted at 2026-09-26 01:05:35 on dist-test-slave-l73v"
server_key: "f85fe5dfbad73706736f0850d49be71c"
server_key_iv: "ee35506c3abc3c3c1d9b6e5aa799a69d"
server_key_version: "encryptionkey@0"
I20260926 01:05:35.982791 9682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:35.999871 9682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:36.000190 9682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:36.000320 9682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:36.000545 9682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:36.001008 9682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:36.001051 9682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.001082 9682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:36.001101 9682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.007552 9682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41811
I20260926 01:05:36.007678 9809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41811 every 8 connection(s)
I20260926 01:05:36.007967 9682 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:05:36.013554 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.008757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41273 (local address 127.0.98.62:42283)
0926 01:05:36.008883 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:36.008885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:36.009065 (+ 180us) server_negotiation.cc:407] Connection header received
0926 01:05:36.009605 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:36.009607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:36.009622 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:36.009671 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:36.010514 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.011034 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.012055 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.012264 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.012335 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:36.013281 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:36.013282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:36.013282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:36.013290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:36.013303 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:36.013307 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:36.013349 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:36.013438 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 01:05:36.013485 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 01:05:36.013712 9812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.008729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42283 (local address 127.0.98.1:41273)
0926 01:05:36.008937 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:36.008980 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:36.009505 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:36.009758 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:36.009763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:36.009774 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:36.010414 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:36.010423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.011131 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.011131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:36.011949 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:36.011952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.012462 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.012463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:36.012601 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:36.013140 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:36.013141 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:36.013171 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:36.013196 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:36.013367 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:36.013370 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:36.013373 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:36.013421 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 01:05:36.013511 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260926 01:05:36.014261 9810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42283
I20260926 01:05:36.014367 9810 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:36.014659 9810 heartbeater.cc:511] Master 127.0.98.62:42283 requested a full tablet report, sending...
I20260926 01:05:36.015184 9622 ts_manager.cc:194] Registered new tserver with Master: 999f56d92b42404ab9cccaf2420c5b73 (127.0.98.1:41811)
I20260926 01:05:36.015857 9622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41273
I20260926 01:05:36.016918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 9682
I20260926 01:05:36.016991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:05:36.017174 392 external_mini_cluster.cc:1428] Setting key d275cff590fd1d2c5945227afeb1cd36
I20260926 01:05:36.018216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42283
--builtin_ntp_servers=127.0.98.20:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 01:05:36.099594 9813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:36.099834 9813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:36.099858 9813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:36.101164 9813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:36.101213 9813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:36.101241 9813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:36.102710 9813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:36.102958 9813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:36.103278 9813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:36.108408 9821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.108425 9818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.108428 9819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:36.108605 9813 server_base.cc:1034] running on GCE node
I20260926 01:05:36.108866 9813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:36.109123 9813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:36.110267 9813 hybrid_clock.cc:648] HybridClock initialized: now 1790384736110234 us; error 50 us; skew 500 ppm
I20260926 01:05:36.111675 9813 webserver.cc:460] Webserver started at http://127.0.98.2:37543/ using document root <none> and password file <none>
I20260926 01:05:36.111922 9813 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:36.111974 9813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:36.112094 9813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:36.113224 9813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "066a472ab33f448ea0479f78b985d5a7"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "26a1e46349de3b7246dc04d7fd6cb471"
server_key_iv: "7b00673b6e4b4d6e961615fdad66993b"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.113564 9813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "066a472ab33f448ea0479f78b985d5a7"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "26a1e46349de3b7246dc04d7fd6cb471"
server_key_iv: "7b00673b6e4b4d6e961615fdad66993b"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.114981 9813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:36.115734 9827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.115890 9813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:36.115986 9813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "066a472ab33f448ea0479f78b985d5a7"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "26a1e46349de3b7246dc04d7fd6cb471"
server_key_iv: "7b00673b6e4b4d6e961615fdad66993b"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.116052 9813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:36.135182 9813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:36.135519 9813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:36.135648 9813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:36.135877 9813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:36.136214 9813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:36.136248 9813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.136278 9813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:36.136300 9813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.141748 9813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42107
I20260926 01:05:36.141788 9940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42107 every 8 connection(s)
I20260926 01:05:36.142127 9813 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:05:36.143183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 9813
I20260926 01:05:36.143261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:05:36.143433 392 external_mini_cluster.cc:1428] Setting key 0c8bce4963f411586cf62efdd7469e5b
I20260926 01:05:36.144325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42283
--builtin_ntp_servers=127.0.98.20:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 01:05:36.148068 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.142883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45833 (local address 127.0.98.62:42283)
0926 01:05:36.142935 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:36.142937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:36.143152 (+ 215us) server_negotiation.cc:407] Connection header received
0926 01:05:36.143739 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:36.143740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:36.143752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:36.143795 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:36.144546 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.145038 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.146468 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.146694 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.146756 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:36.147760 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:36.147762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:36.147762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:36.147768 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:36.147784 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:36.147789 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:36.147834 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:36.147948 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 01:05:36.147982 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:05:36.148281 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.142859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42283 (local address 127.0.98.2:45833)
0926 01:05:36.143041 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:36.143075 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:36.143644 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:36.143875 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:36.143879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:36.143891 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:36.144472 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:36.144478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.145135 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.145136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:36.146359 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:36.146362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.146783 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.146784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:36.146941 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:36.147598 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:36.147598 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:36.147640 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:36.147666 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:36.147876 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:36.147880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:36.147883 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:36.147943 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 01:05:36.148058 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260926 01:05:36.148758 9941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42283
I20260926 01:05:36.148852 9941 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:36.149035 9941 heartbeater.cc:511] Master 127.0.98.62:42283 requested a full tablet report, sending...
I20260926 01:05:36.149402 9622 ts_manager.cc:194] Registered new tserver with Master: 066a472ab33f448ea0479f78b985d5a7 (127.0.98.2:42107)
I20260926 01:05:36.149762 9622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45833
W20260926 01:05:36.227461 9944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:36.227701 9944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:36.227730 9944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:36.228974 9944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:36.229017 9944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:36.229051 9944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:36.230528 9944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:36.230794 9944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:36.231101 9944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:05:36.236218 9944 server_base.cc:1034] running on GCE node
W20260926 01:05:36.236181 9949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.236181 9952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.236236 9950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:36.236629 9944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:36.236879 9944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:36.238021 9944 hybrid_clock.cc:648] HybridClock initialized: now 1790384736237997 us; error 51 us; skew 500 ppm
I20260926 01:05:36.239344 9944 webserver.cc:460] Webserver started at http://127.0.98.3:34167/ using document root <none> and password file <none>
I20260926 01:05:36.239585 9944 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:36.239632 9944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:36.239760 9944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:36.240886 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "e785be13d51847428988f7252159439a"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "63d90e8858bb5ccc977972d12fa92504"
server_key_iv: "f2cd81b3914409e945e85aa678b74802"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.241231 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e785be13d51847428988f7252159439a"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "63d90e8858bb5ccc977972d12fa92504"
server_key_iv: "f2cd81b3914409e945e85aa678b74802"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.242552 9944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:36.243232 9958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.243386 9944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:36.243467 9944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "e785be13d51847428988f7252159439a"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "63d90e8858bb5ccc977972d12fa92504"
server_key_iv: "f2cd81b3914409e945e85aa678b74802"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.243564 9944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:36.257129 9944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:36.257443 9944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:36.257581 9944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:36.257802 9944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:36.258121 9944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:36.258155 9944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.258184 9944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:36.258208 9944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.263619 9944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38157
I20260926 01:05:36.263693 10071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38157 every 8 connection(s)
I20260926 01:05:36.263989 9944 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:05:36.269009 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.264773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59917 (local address 127.0.98.62:42283)
0926 01:05:36.264854 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:36.264857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:36.265051 (+ 194us) server_negotiation.cc:407] Connection header received
0926 01:05:36.265568 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:36.265570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:36.265580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:36.265631 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:36.266277 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.266845 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.267809 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.267992 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.268038 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:36.268796 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:36.268797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:36.268797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:36.268802 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:36.268815 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:36.268819 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:36.268856 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:36.268914 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 01:05:36.268939 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 01:05:36.269207 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.264705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42283 (local address 127.0.98.3:59917)
0926 01:05:36.264901 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:36.264939 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:36.265462 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:36.265687 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:36.265690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:36.265698 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:36.266192 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:36.266197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.266926 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.266927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:36.267742 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:36.267744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:36.268052 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:36.268052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:36.268158 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:36.268659 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:36.268660 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:36.268689 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:36.268714 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:36.268858 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:36.268861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:36.268863 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:05:36.268908 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:05:36.268987 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260926 01:05:36.269440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 9944
I20260926 01:05:36.269526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:05:36.269735 392 external_mini_cluster.cc:1428] Setting key 49f324a2729176e6bd5358fb05830f2e
I20260926 01:05:36.269723 10072 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42283
I20260926 01:05:36.269810 10072 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:36.269986 10072 heartbeater.cc:511] Master 127.0.98.62:42283 requested a full tablet report, sending...
I20260926 01:05:36.270421 9621 ts_manager.cc:194] Registered new tserver with Master: e785be13d51847428988f7252159439a (127.0.98.3:38157)
I20260926 01:05:36.270792 9621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59917
I20260926 01:05:36.281672 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 01:05:36.283692 9662 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:36.283189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.0.98.62:42283)
0926 01:05:36.283250 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:36.283252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:36.283280 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:05:36.283350 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:36.283350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:36.283363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:36.283419 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:36.283613 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50204: BlockingRecv error: recv got EOF from 127.0.0.1:50204 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:05:36.283720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 9682
I20260926 01:05:36.287639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 9813
I20260926 01:05:36.291290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 9944
I20260926 01:05:36.295468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 9591
2026-09-26T01:05:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (616 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T01:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:36Z Disabled control of system clock
I20260926 01:05:36.320712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39139
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:05:36.401929 10083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:36.402174 10083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:36.402197 10083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:36.403548 10083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:36.403606 10083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:36.403620 10083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:36.403633 10083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:36.403646 10083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:36.405143 10083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:36.405423 10083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:36.405706 10083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:36.410539 10091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:36.410647 10083 server_base.cc:1034] running on GCE node
W20260926 01:05:36.410576 10088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.410574 10089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:36.410964 10083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:36.411257 10083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:36.412452 10083 hybrid_clock.cc:648] HybridClock initialized: now 1790384736412430 us; error 34 us; skew 500 ppm
I20260926 01:05:36.413883 10083 webserver.cc:460] Webserver started at http://127.0.98.62:36599/ using document root <none> and password file <none>
I20260926 01:05:36.414112 10083 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:36.414175 10083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:36.414323 10083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:36.415668 10083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "66637e62421948a1a270487ec8501b0c"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "4acc0e964eeb225c0f1ef83976289097"
server_key_iv: "b2801bf929f82ca00eb49167980486c1"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.416051 10083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "66637e62421948a1a270487ec8501b0c"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "4acc0e964eeb225c0f1ef83976289097"
server_key_iv: "b2801bf929f82ca00eb49167980486c1"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.417519 10083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:05:36.418349 10097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.418553 10083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:36.418648 10083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "66637e62421948a1a270487ec8501b0c"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "4acc0e964eeb225c0f1ef83976289097"
server_key_iv: "b2801bf929f82ca00eb49167980486c1"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.418716 10083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:36.422145 10083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:36.422463 10083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:36.422603 10083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:36.426368 10083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39139
I20260926 01:05:36.426421 10149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39139 every 8 connection(s)
I20260926 01:05:36.426756 10083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:36.427448 10150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:36.430434 10150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c: Bootstrap starting.
I20260926 01:05:36.431049 10150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:36.431325 10150 log.cc:824] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:36.432132 10150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c: No bootstrap required, opened a new log
I20260926 01:05:36.433833 10150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66637e62421948a1a270487ec8501b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } }
I20260926 01:05:36.433997 10150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:36.434020 10150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66637e62421948a1a270487ec8501b0c, State: Initialized, Role: FOLLOWER
I20260926 01:05:36.434161 10150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66637e62421948a1a270487ec8501b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } }
I20260926 01:05:36.434235 10150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:36.434258 10150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:36.434288 10150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:36.434945 10150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66637e62421948a1a270487ec8501b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } }
I20260926 01:05:36.435050 10150 leader_election.cc:302] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66637e62421948a1a270487ec8501b0c; no voters:
I20260926 01:05:36.435231 10150 leader_election.cc:288] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:36.435292 10153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:36.435465 10153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [term 1 LEADER]: Becoming Leader. State: Replica: 66637e62421948a1a270487ec8501b0c, State: Running, Role: LEADER
I20260926 01:05:36.435513 10150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:36.435638 10153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66637e62421948a1a270487ec8501b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } }
I20260926 01:05:36.435931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 10083
I20260926 01:05:36.436019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:36.436055 10154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66637e62421948a1a270487ec8501b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66637e62421948a1a270487ec8501b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } }
I20260926 01:05:36.436126 10154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:36.436214 392 external_mini_cluster.cc:1428] Setting key 60e624bc64c108762534d2135c02babd
I20260926 01:05:36.436260 10155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66637e62421948a1a270487ec8501b0c. Latest consensus state: current_term: 1 leader_uuid: "66637e62421948a1a270487ec8501b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66637e62421948a1a270487ec8501b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39139 } } }
I20260926 01:05:36.436312 10155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:36.436327 10162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:36.436828 10162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:36.438443 10162 catalog_manager.cc:1347] Generated new cluster ID: ca0bfa3191ba4ee38ec1e280acbf0ce7
I20260926 01:05:36.438504 10162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:36.439054 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.436825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.0.98.62:39139)
0926 01:05:36.437042 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:36.437142 (+ 100us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:36.437215 (+ 73us) server_negotiation.cc:407] Connection header received
0926 01:05:36.437475 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:36.437484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:36.437546 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:36.437697 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:36.438612 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:36.438615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:36.438616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:36.438620 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:36.438664 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:36.438677 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:36.438727 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:36.438817 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 01:05:36.438879 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260926 01:05:36.461863 10162 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:36.462057 10162 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:36.476311 10162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66637e62421948a1a270487ec8501b0c: Generated new TSK 0
I20260926 01:05:36.476531 10162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:05:36.486369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39139
--builtin_ntp_servers=127.0.98.20:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:05:36.570174 10174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:36.570436 10174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:36.570472 10174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:36.571858 10174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:36.571913 10174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:36.571951 10174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:36.573403 10174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:36.573668 10174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:36.573969 10174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:36.578779 10182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.578784 10180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.578796 10179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:36.579048 10174 server_base.cc:1034] running on GCE node
I20260926 01:05:36.579274 10174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:36.579524 10174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:36.580700 10174 hybrid_clock.cc:648] HybridClock initialized: now 1790384736580685 us; error 25 us; skew 500 ppm
I20260926 01:05:36.582012 10174 webserver.cc:460] Webserver started at http://127.0.98.1:44445/ using document root <none> and password file <none>
I20260926 01:05:36.582260 10174 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:36.582307 10174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:36.582499 10174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:36.583565 10174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "398c54a82e9c49a99cc30c81020849b6"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "f536f0e65ec320e816f51854212a007c"
server_key_iv: "19946e523c965d9571bd2a789aa940e3"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.583894 10174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "398c54a82e9c49a99cc30c81020849b6"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "f536f0e65ec320e816f51854212a007c"
server_key_iv: "19946e523c965d9571bd2a789aa940e3"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.585160 10174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:05:36.586004 10188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.586210 10174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:36.586299 10174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "398c54a82e9c49a99cc30c81020849b6"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "f536f0e65ec320e816f51854212a007c"
server_key_iv: "19946e523c965d9571bd2a789aa940e3"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.586362 10174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:36.589982 10174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:36.590245 10174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:36.590356 10174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:36.590591 10174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:36.590880 10174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:36.590914 10174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.590937 10174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:36.590957 10174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.597077 10174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43671
I20260926 01:05:36.597146 10301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43671 every 8 connection(s)
I20260926 01:05:36.597478 10174 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:05:36.600569 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.598297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59369 (local address 127.0.98.62:39139)
0926 01:05:36.598376 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:36.598378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:36.598549 (+ 171us) server_negotiation.cc:407] Connection header received
0926 01:05:36.599080 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:36.599081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:36.599090 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:36.599165 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:36.600344 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:36.600345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:36.600346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:36.600350 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:36.600368 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:36.600372 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:36.600434 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:36.600495 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 01:05:36.600509 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 01:05:36.600752 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.598219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39139 (local address 127.0.98.1:59369)
0926 01:05:36.598428 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:36.598470 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:36.598953 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:36.599224 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:36.599227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:36.599236 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:36.600164 (+ 928us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:36.600165 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:36.600197 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:36.600230 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:36.600436 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:36.600439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:36.600442 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:36.600497 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:05:36.600549 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260926 01:05:36.601398 10302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39139
I20260926 01:05:36.601430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 10174
I20260926 01:05:36.601504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:05:36.601514 10302 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:36.601713 392 external_mini_cluster.cc:1428] Setting key df1cdacc74e90ac23cdf327e0b002a56
I20260926 01:05:36.601715 10302 heartbeater.cc:511] Master 127.0.98.62:39139 requested a full tablet report, sending...
I20260926 01:05:36.602274 10113 ts_manager.cc:194] Registered new tserver with Master: 398c54a82e9c49a99cc30c81020849b6 (127.0.98.1:43671)
I20260926 01:05:36.602713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39139
--builtin_ntp_servers=127.0.98.20:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:05:36.685338 10305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:36.685578 10305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:36.685604 10305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:36.686877 10305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:36.686925 10305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:36.686952 10305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:36.688366 10305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:36.688577 10305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:36.688853 10305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:36.694046 10310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.694049 10313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.694049 10311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:36.694211 10305 server_base.cc:1034] running on GCE node
I20260926 01:05:36.694497 10305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:36.694777 10305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:36.695959 10305 hybrid_clock.cc:648] HybridClock initialized: now 1790384736695938 us; error 33 us; skew 500 ppm
I20260926 01:05:36.697393 10305 webserver.cc:460] Webserver started at http://127.0.98.2:44899/ using document root <none> and password file <none>
I20260926 01:05:36.697644 10305 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:36.697698 10305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:36.697837 10305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:36.698983 10305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "c9f7765842f04b8fa6065056d7b80cfe"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "c292b5056b03e96f914fefdbc5fc58ad"
server_key_iv: "59aee11c7692cf6dcaded19003a637f3"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.699332 10305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c9f7765842f04b8fa6065056d7b80cfe"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "c292b5056b03e96f914fefdbc5fc58ad"
server_key_iv: "59aee11c7692cf6dcaded19003a637f3"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.700650 10305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:36.701467 10319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.701644 10305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:36.701728 10305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "c9f7765842f04b8fa6065056d7b80cfe"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "c292b5056b03e96f914fefdbc5fc58ad"
server_key_iv: "59aee11c7692cf6dcaded19003a637f3"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.701793 10305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:36.705222 10305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:36.705489 10305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:36.705627 10305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:36.705824 10305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:36.706130 10305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:36.706164 10305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.706192 10305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:36.706215 10305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.711913 10305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39055
I20260926 01:05:36.711957 10432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39055 every 8 connection(s)
I20260926 01:05:36.712291 10305 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:05:36.715312 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.712973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45449 (local address 127.0.98.62:39139)
0926 01:05:36.713035 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:36.713036 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:36.713275 (+ 239us) server_negotiation.cc:407] Connection header received
0926 01:05:36.713856 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:36.713857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:36.713863 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:36.713922 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:36.715093 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:36.715094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:36.715094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:36.715098 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:36.715114 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:36.715119 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:36.715176 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:36.715239 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 01:05:36.715251 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:05:36.715497 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.712971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39139 (local address 127.0.98.2:45449)
0926 01:05:36.713148 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:36.713184 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:36.713710 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:36.713981 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:36.713985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:36.713994 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:36.714923 (+ 929us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:36.714924 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:36.714956 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:36.714988 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:36.715180 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:36.715184 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:36.715186 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:05:36.715241 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:05:36.715293 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260926 01:05:36.716023 10433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39139
I20260926 01:05:36.716128 10433 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:36.716280 10433 heartbeater.cc:511] Master 127.0.98.62:39139 requested a full tablet report, sending...
I20260926 01:05:36.716641 10113 ts_manager.cc:194] Registered new tserver with Master: c9f7765842f04b8fa6065056d7b80cfe (127.0.98.2:39055)
I20260926 01:05:36.717449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 10305
I20260926 01:05:36.717530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:05:36.717710 392 external_mini_cluster.cc:1428] Setting key e8b89f2f4129c345bb65c5f1efd67287
I20260926 01:05:36.718928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39139
--builtin_ntp_servers=127.0.98.20:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 01:05:36.805369 10436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:36.805615 10436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:36.805640 10436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:36.806924 10436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:36.806965 10436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:36.806993 10436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:36.808359 10436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:36.808615 10436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:36.808903 10436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:36.813918 10444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.813891 10442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:36.814080 10441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:36.814028 10436 server_base.cc:1034] running on GCE node
I20260926 01:05:36.814334 10436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:36.814596 10436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:36.815740 10436 hybrid_clock.cc:648] HybridClock initialized: now 1790384736815728 us; error 29 us; skew 500 ppm
I20260926 01:05:36.816933 10436 webserver.cc:460] Webserver started at http://127.0.98.3:32877/ using document root <none> and password file <none>
I20260926 01:05:36.817152 10436 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:36.817198 10436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:36.817323 10436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:36.818490 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "4bd02e7e097447b7b412c719e786ea64"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "6d6dcd12a65c84d2a0c018d957a7fed6"
server_key_iv: "99c3433e9488a252bb095ac137f9ca52"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.818837 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4bd02e7e097447b7b412c719e786ea64"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "6d6dcd12a65c84d2a0c018d957a7fed6"
server_key_iv: "99c3433e9488a252bb095ac137f9ca52"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.820210 10436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:36.821131 10450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.821281 10436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:36.821362 10436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "4bd02e7e097447b7b412c719e786ea64"
format_stamp: "Formatted at 2026-09-26 01:05:36 on dist-test-slave-l73v"
server_key: "6d6dcd12a65c84d2a0c018d957a7fed6"
server_key_iv: "99c3433e9488a252bb095ac137f9ca52"
server_key_version: "encryptionkey@0"
I20260926 01:05:36.821434 10436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:36.824962 10436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:36.825241 10436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:36.825364 10436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:36.825590 10436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:36.825897 10436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:36.825929 10436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.825961 10436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:36.825984 10436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:36.831812 10436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34935
I20260926 01:05:36.831877 10563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34935 every 8 connection(s)
I20260926 01:05:36.832197 10436 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:05:36.835381 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.832941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40229 (local address 127.0.98.62:39139)
0926 01:05:36.833001 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:36.833003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:36.833238 (+ 235us) server_negotiation.cc:407] Connection header received
0926 01:05:36.833773 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:36.833775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:36.833782 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:36.833835 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:36.835112 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:36.835113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:36.835113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:05:36.835119 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:36.835133 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:05:36.835138 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:36.835215 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:36.835288 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 01:05:36.835303 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:05:36.835592 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:36.832885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39139 (local address 127.0.98.3:40229)
0926 01:05:36.833116 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:36.833154 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:36.833668 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:36.833906 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:36.833910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:36.833920 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:36.834937 (+ 1017us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:05:36.834938 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:36.834970 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:05:36.835001 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:36.835215 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:36.835220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:36.835224 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:36.835293 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 01:05:36.835377 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260926 01:05:36.836102 10564 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39139
I20260926 01:05:36.836222 10564 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:36.836380 10564 heartbeater.cc:511] Master 127.0.98.62:39139 requested a full tablet report, sending...
I20260926 01:05:36.836774 10113 ts_manager.cc:194] Registered new tserver with Master: 4bd02e7e097447b7b412c719e786ea64 (127.0.98.3:34935)
I20260926 01:05:36.840154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 10436
I20260926 01:05:36.840240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:05:36.840389 392 external_mini_cluster.cc:1428] Setting key 4747e7388c76aef88aea32f37d8dd4fc
I20260926 01:05:36.841355 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 01:05:36.843143 10170 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:05:36.842681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.0.98.62:39139)
0926 01:05:36.842750 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:36.842752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:36.842825 (+ 73us) server_negotiation.cc:407] Connection header received
0926 01:05:36.842893 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:36.842894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:36.842899 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:36.842947 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:36.843070 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51578: BlockingRecv error: recv got EOF from 127.0.0.1:51578 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:05:36.843216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 10174
I20260926 01:05:36.847569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 10305
I20260926 01:05:36.852643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 10436
I20260926 01:05:36.856868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 10083
2026-09-26T01:05:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (561 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:05:36 dist-test-slave-l73v krb5kdc[10574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:05:36 dist-test-slave-l73v krb5kdc[10574](info): set up 2 sockets
Sep 26 01:05:36 dist-test-slave-l73v krb5kdc[10574](info): commencing operation
krb5kdc: starting...
W20260926 01:05:37.807111 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:05:37 dist-test-slave-l73v krb5kdc[10574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:05:37.930366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43325
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:38.010885 10587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:38.011127 10587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:38.011150 10587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:38.012653 10587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:05:38.012744 10587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:38.012760 10587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:38.012773 10587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:05:38.012784 10587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:05:38.014606 10587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:38.014902 10587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:38.015136 10587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:38.020135 10593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:38.020135 10592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:38.020200 10595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:38.020545 10587 server_base.cc:1034] running on GCE node
I20260926 01:05:38.020782 10587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:38.021138 10587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:38.022307 10587 hybrid_clock.cc:648] HybridClock initialized: now 1790384738022277 us; error 45 us; skew 500 ppm
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[10574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:38.025051 10587 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:05:38.025583 10587 webserver.cc:460] Webserver started at http://127.0.98.62:35871/ using document root <none> and password file <none>
I20260926 01:05:38.025796 10587 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:38.025846 10587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:38.025974 10587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:38.027125 10587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "186a57cf595244bbbf82cdf50da0c9e1"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "c179d41444a3ae9d21f328faeb1fe532"
server_key_iv: "f49b6cf30166996cb7e22680e77e01d0"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.027472 10587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "186a57cf595244bbbf82cdf50da0c9e1"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "c179d41444a3ae9d21f328faeb1fe532"
server_key_iv: "f49b6cf30166996cb7e22680e77e01d0"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.028842 10587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:38.029685 10602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:38.029968 10587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:38.030074 10587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "186a57cf595244bbbf82cdf50da0c9e1"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "c179d41444a3ae9d21f328faeb1fe532"
server_key_iv: "f49b6cf30166996cb7e22680e77e01d0"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.030143 10587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:38.046990 10587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:38.049223 10587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:38.049377 10587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:38.053694 10587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43325
I20260926 01:05:38.053743 10654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43325 every 8 connection(s)
I20260926 01:05:38.054091 10587 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:05:38.055156 10655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:38.055541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 10587
I20260926 01:05:38.055635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:05:38.055805 392 external_mini_cluster.cc:1428] Setting key eb53fe3e6e8984b70bd902d0c135cf18
I20260926 01:05:38.057735 10655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1: Bootstrap starting.
I20260926 01:05:38.058333 10655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:38.058627 10655 log.cc:824] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:38.059293 10655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1: No bootstrap required, opened a new log
I20260926 01:05:38.061113 10655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a57cf595244bbbf82cdf50da0c9e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43325 } }
I20260926 01:05:38.061272 10655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:38.061306 10655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186a57cf595244bbbf82cdf50da0c9e1, State: Initialized, Role: FOLLOWER
I20260926 01:05:38.061445 10655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a57cf595244bbbf82cdf50da0c9e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43325 } }
I20260926 01:05:38.061532 10655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:05:38.061565 10655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:05:38.061612 10655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[10574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:38.062366 10655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a57cf595244bbbf82cdf50da0c9e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43325 } }
I20260926 01:05:38.062502 10655 leader_election.cc:302] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 186a57cf595244bbbf82cdf50da0c9e1; no voters:
I20260926 01:05:38.062686 10655 leader_election.cc:288] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:05:38.062754 10660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:38.062945 10660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 186a57cf595244bbbf82cdf50da0c9e1, State: Running, Role: LEADER
I20260926 01:05:38.063048 10655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:05:38.063086 10660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a57cf595244bbbf82cdf50da0c9e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43325 } }
I20260926 01:05:38.063489 10662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "186a57cf595244bbbf82cdf50da0c9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a57cf595244bbbf82cdf50da0c9e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43325 } } }
I20260926 01:05:38.063482 10660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 186a57cf595244bbbf82cdf50da0c9e1. Latest consensus state: current_term: 1 leader_uuid: "186a57cf595244bbbf82cdf50da0c9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a57cf595244bbbf82cdf50da0c9e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43325 } } }
I20260926 01:05:38.063573 10662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:38.063580 10660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1 [sys.catalog]: This master's current role is: LEADER
I20260926 01:05:38.063759 10668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:05:38.064288 10668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:05:38.066606 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.056369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.0.98.62:43325)
0926 01:05:38.056549 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.056557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.056584 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:05:38.056990 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.056997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.057044 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.057178 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:38.057820 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.058886 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.059679 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.060108 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.060209 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.062799 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:38.062804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:38.062804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:38.062819 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:38.064911 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:38.065337 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:38.065341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:38.065341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:38.065452 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:38.065662 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:38.065663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:38.065663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:38.065968 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:38.066117 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.066215 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.066368 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260926 01:05:38.066813 10668 catalog_manager.cc:1347] Generated new cluster ID: 6590428e4d2f43b8bc0f28332db53ad2
I20260926 01:05:38.067013 10668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:05:38.077867 10668 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:05:38.078441 10668 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:05:38.082197 10668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1: Generated new TSK 0
I20260926 01:05:38.082376 10668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:05:38.113750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43325
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:38.197846 10681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:38.198100 10681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:38.198137 10681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:38.199543 10681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:38.199616 10681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:38.199658 10681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:05:38.201375 10681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:38.201714 10681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:38.201954 10681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:05:38.207151 10681 server_base.cc:1034] running on GCE node
W20260926 01:05:38.207253 10687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:38.207290 10686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:38.207144 10689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:38.207618 10681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:38.207901 10681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:38.209066 10681 hybrid_clock.cc:648] HybridClock initialized: now 1790384738209050 us; error 35 us; skew 500 ppm
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[10574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:38.211836 10681 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:05:38.212291 10681 webserver.cc:460] Webserver started at http://127.0.98.1:40627/ using document root <none> and password file <none>
I20260926 01:05:38.212488 10681 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:38.212530 10681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:38.212626 10681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:38.213691 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "3dd15551d42441088e7cec7cfc16d140"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "c5fab93f01f82c9899a1bff264327e83"
server_key_iv: "4827122099a1985a07020b027d775df4"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.214007 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3dd15551d42441088e7cec7cfc16d140"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "c5fab93f01f82c9899a1bff264327e83"
server_key_iv: "4827122099a1985a07020b027d775df4"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.215732 10681 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 01:05:38.216558 10696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:38.216747 10681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:05:38.216835 10681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "3dd15551d42441088e7cec7cfc16d140"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "c5fab93f01f82c9899a1bff264327e83"
server_key_iv: "4827122099a1985a07020b027d775df4"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.216920 10681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:38.242661 10681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:38.244705 10681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:38.245087 10681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:38.245455 10681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:38.246035 10681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:38.246093 10681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:38.246119 10681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:38.246129 10681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:38.251857 10681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40999
I20260926 01:05:38.251919 10809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40999 every 8 connection(s)
I20260926 01:05:38.252228 10681 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[10574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:38.259442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 10681
I20260926 01:05:38.259547 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:05:38.259756 392 external_mini_cluster.cc:1428] Setting key efd093152bd206b2b38b95d84e1854a9
I20260926 01:05:38.262854 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.253001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49883 (local address 127.0.98.62:43325)
0926 01:05:38.253080 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.253082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.253239 (+ 157us) server_negotiation.cc:407] Connection header received
0926 01:05:38.253718 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.253719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.253728 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.253771 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:38.254737 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.255258 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.256359 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.256590 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.256654 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.259359 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:38.259361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:38.259362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:38.259371 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:38.261256 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:38.261747 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:38.261748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:38.261748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:38.261813 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:38.262118 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:38.262118 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:38.262119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:38.262241 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:38.262292 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.262697 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.262786 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 01:05:38.263175 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.252966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43325 (local address 127.0.98.1:49883)
0926 01:05:38.253122 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:38.253158 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:38.253624 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:38.253826 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:38.253829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:38.254420 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:38.254616 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.254622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.255364 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.255365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.256224 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.256227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.256682 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.256683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.256826 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.257516 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:38.257525 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:38.259229 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:38.261362 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:38.261372 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:38.261374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:38.261674 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:38.261893 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:38.261894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:38.261894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:38.262053 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:38.262296 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:38.262303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:38.262521 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 01:05:38.262666 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 01:05:38.262857 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260926 01:05:38.263617 10810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43325
I20260926 01:05:38.263710 10810 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:38.263914 10810 heartbeater.cc:511] Master 127.0.98.62:43325 requested a full tablet report, sending...
I20260926 01:05:38.264406 10619 ts_manager.cc:194] Registered new tserver with Master: 3dd15551d42441088e7cec7cfc16d140 (127.0.98.1:40999)
I20260926 01:05:38.265035 10619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49883
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:05:38.289613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43325
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:38.372435 10815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:38.372681 10815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:38.372720 10815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:38.374042 10815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:38.374104 10815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:38.374145 10815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:05:38.375867 10815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:38.376157 10815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:38.376395 10815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:38.381485 10821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:38.381556 10823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:38.381618 10820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:38.381492 10815 server_base.cc:1034] running on GCE node
I20260926 01:05:38.381913 10815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:38.382157 10815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:38.383313 10815 hybrid_clock.cc:648] HybridClock initialized: now 1790384738383299 us; error 28 us; skew 500 ppm
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[10574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:38.385977 10815 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:05:38.386520 10815 webserver.cc:460] Webserver started at http://127.0.98.2:34969/ using document root <none> and password file <none>
I20260926 01:05:38.386736 10815 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:38.386782 10815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:38.386927 10815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:38.388018 10815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "f5bb93b24d944765b4bcfe6780550fff"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "15a8d72389cc8adf42d2a547c5092ffd"
server_key_iv: "77e5121f38acbee3e52e5a0a7e389bf6"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.388351 10815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f5bb93b24d944765b4bcfe6780550fff"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "15a8d72389cc8adf42d2a547c5092ffd"
server_key_iv: "77e5121f38acbee3e52e5a0a7e389bf6"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.389671 10815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:05:38.390586 10830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:38.390792 10815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:05:38.390883 10815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "f5bb93b24d944765b4bcfe6780550fff"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "15a8d72389cc8adf42d2a547c5092ffd"
server_key_iv: "77e5121f38acbee3e52e5a0a7e389bf6"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.390976 10815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:38.410597 10815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:38.412791 10815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:38.412930 10815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:38.413172 10815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:38.413488 10815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:38.413523 10815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:38.413545 10815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:38.413556 10815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:38.419644 10815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46329
I20260926 01:05:38.419703 10943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46329 every 8 connection(s)
I20260926 01:05:38.420050 10815 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:05:38.424942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 10815
I20260926 01:05:38.425058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:05:38.425261 392 external_mini_cluster.cc:1428] Setting key 3f82fd09a3e6a0f568f88f6def2305d7
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[10574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:38.430752 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.420849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57377 (local address 127.0.98.62:43325)
0926 01:05:38.420904 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.420906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.421111 (+ 205us) server_negotiation.cc:407] Connection header received
0926 01:05:38.421591 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.421592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.421605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.421668 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:38.422716 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.423225 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.424337 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.424688 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.424762 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.427532 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:38.427533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:38.427534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:38.427542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:38.429088 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:38.429599 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:38.429601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:38.429601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:38.429666 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:38.429964 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:38.429965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:38.429965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:38.430114 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:38.430192 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.430561 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.430651 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:05:38.430879 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.420768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43325 (local address 127.0.98.2:57377)
0926 01:05:38.420961 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:38.421003 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:38.421502 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:38.421733 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:38.421737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:38.422314 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:38.422599 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.422605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.423344 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.423345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.424181 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.424183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.424812 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.424813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.424976 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.425634 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:38.425643 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:38.427418 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:38.429213 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:38.429220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:38.429222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:38.429515 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:38.429763 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:38.429764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:38.429765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:38.429892 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:38.430205 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:38.430211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:38.430374 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 01:05:38.430522 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 01:05:38.430668 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":117,"threads_started":1}
I20260926 01:05:38.431362 10944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43325
I20260926 01:05:38.431458 10944 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:38.431632 10944 heartbeater.cc:511] Master 127.0.98.62:43325 requested a full tablet report, sending...
I20260926 01:05:38.432018 10619 ts_manager.cc:194] Registered new tserver with Master: f5bb93b24d944765b4bcfe6780550fff (127.0.98.2:46329)
I20260926 01:05:38.432518 10619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57377
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:05:38.453595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43325
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:05:38.536096 10949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:05:38.536334 10949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:05:38.536358 10949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:05:38.537585 10949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:05:38.537637 10949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:05:38.537667 10949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:05:38.539212 10949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:05:38.539489 10949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:05:38.539705 10949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:05:38.544615 10957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:38.544710 10955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:05:38.544837 10954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:05:38.544724 10949 server_base.cc:1034] running on GCE node
I20260926 01:05:38.545045 10949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:05:38.545317 10949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:05:38.546468 10949 hybrid_clock.cc:648] HybridClock initialized: now 1790384738546448 us; error 29 us; skew 500 ppm
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[10574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:05:38.549086 10949 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:05:38.549546 10949 webserver.cc:460] Webserver started at http://127.0.98.3:43085/ using document root <none> and password file <none>
I20260926 01:05:38.549755 10949 fs_manager.cc:359] Metadata directory not provided
I20260926 01:05:38.549806 10949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:05:38.549932 10949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:05:38.551009 10949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "1cb79d784ebc4508a430915df2ffb29e"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "bdb8726e67a308b5722efe2f89517a7f"
server_key_iv: "383aea536c63e98daea1a0b6e793dab2"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.551335 10949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1cb79d784ebc4508a430915df2ffb29e"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "bdb8726e67a308b5722efe2f89517a7f"
server_key_iv: "383aea536c63e98daea1a0b6e793dab2"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.552567 10949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:05:38.553452 10964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:38.553606 10949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:05:38.553696 10949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "1cb79d784ebc4508a430915df2ffb29e"
format_stamp: "Formatted at 2026-09-26 01:05:38 on dist-test-slave-l73v"
server_key: "bdb8726e67a308b5722efe2f89517a7f"
server_key_iv: "383aea536c63e98daea1a0b6e793dab2"
server_key_version: "encryptionkey@0"
I20260926 01:05:38.553771 10949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:05:38.567862 10949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:05:38.569991 10949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:05:38.570132 10949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:05:38.570354 10949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:05:38.570737 10949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:05:38.570771 10949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:38.570801 10949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:05:38.570823 10949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:05:38.576247 10949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44159
I20260926 01:05:38.576288 11077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44159 every 8 connection(s)
I20260926 01:05:38.576614 10949 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:05:38.578577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 10949
I20260926 01:05:38.578673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:05:38.578861 392 external_mini_cluster.cc:1428] Setting key 979258444d89229f5804d405a37b5055
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[10574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:05:38.585942 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.577349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42533 (local address 127.0.98.62:43325)
0926 01:05:38.577405 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.577407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.577584 (+ 177us) server_negotiation.cc:407] Connection header received
0926 01:05:38.578123 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.578124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.578138 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.578191 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:38.579428 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.580004 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.581056 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.581255 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.581316 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.583554 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:38.583556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:38.583556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:38.583563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:38.584816 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:38.585196 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:38.585197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:38.585197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:38.585244 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:38.585448 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:38.585449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:38.585449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:38.585550 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:38.585593 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.585812 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.585873 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:05:38.586177 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.577322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43325 (local address 127.0.98.3:42533)
0926 01:05:38.577467 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:38.577506 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:38.578022 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:38.578269 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:38.578274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:38.579074 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:05:38.579323 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.579330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.580108 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.580109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.580964 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.580967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.581322 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.581322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.581423 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.582018 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:05:38.582029 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:05:38.583454 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:05:38.584909 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:38.584916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:38.584918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:38.585136 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:38.585296 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:05:38.585297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:05:38.585297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:05:38.585403 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:05:38.585593 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:05:38.585596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:05:38.585718 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 01:05:38.585796 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:05:38.585950 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260926 01:05:38.586650 11078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43325
I20260926 01:05:38.586742 11078 heartbeater.cc:464] Registering TS with master...
I20260926 01:05:38.586925 11078 heartbeater.cc:511] Master 127.0.98.62:43325 requested a full tablet report, sending...
I20260926 01:05:38.587283 10619 ts_manager.cc:194] Registered new tserver with Master: 1cb79d784ebc4508a430915df2ffb29e (127.0.98.3:44159)
I20260926 01:05:38.587646 10619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42533
I20260926 01:05:38.590566 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:05:38.597945 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.591827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.0.98.62:43325)
0926 01:05:38.591883 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.591885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.591952 (+ 67us) server_negotiation.cc:407] Connection header received
0926 01:05:38.592006 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.592006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.592015 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.592061 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:05:38.592586 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.593073 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.593791 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.593999 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.594056 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.595420 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:05:38.595421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:05:38.595421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:05:38.595429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:05:38.596741 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:38.597102 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:38.597103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:38.597103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:38.597167 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:05:38.597449 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:05:38.597450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:05:38.597450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:05:38.597617 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:05:38.597704 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.597807 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.597875 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:05:38.599777 10619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:05:38.601351 10619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:05:38.607769 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.603849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.0.98.1:40999)
0926 01:05:38.604318 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.604322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.604331 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:38.604365 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.604367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.604380 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.604605 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:38.604666 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.605337 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.606828 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.607361 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.607425 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.607486 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.607632 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.607676 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":68,"threads_started":1}
I20260926 01:05:38.607760 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.603658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40999 (local address 127.0.0.1:59444)
0926 01:05:38.603895 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:38.603938 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:38.604002 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:38.604472 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:38.604473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:38.604475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:38.604537 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.604538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.605437 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.605438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.606737 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.606739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.607442 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.607443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.607531 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.607544 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 01:05:38.607607 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:05:38.607666 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":31,"thread_start_us":68,"threads_started":1}
I20260926 01:05:38.607968 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.604066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.0.98.3:44159)
0926 01:05:38.604218 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.604222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.604232 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:38.604270 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.604272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.604286 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.604370 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:38.604564 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.605611 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.606780 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.607460 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.607530 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.607584 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.607835 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.607883 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260926 01:05:38.607966 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.603674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46329 (local address 127.0.0.1:35140)
0926 01:05:38.604014 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:38.604024 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:38.604077 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:38.604305 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:38.604308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:38.604310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:38.604386 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.604388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.605703 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.605704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.606893 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.606895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.607770 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.607771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.607822 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.607825 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:38.607869 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:05:38.607901 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":61,"thread_start_us":88,"threads_started":1}
I20260926 01:05:38.608076 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.603870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44159 (local address 127.0.0.1:46706)
0926 01:05:38.604162 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:38.604170 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:38.604206 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:38.604439 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:38.604439 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:38.604441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:38.604489 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.604494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.605711 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.605712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.606704 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.606706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.607618 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.607619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.607688 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.607692 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:05:38.607734 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 01:05:38.607776 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":41,"thread_start_us":71,"threads_started":1}
I20260926 01:05:38.608213 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.603914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.0.98.2:46329)
0926 01:05:38.604120 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.604125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.604134 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:05:38.604188 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.604190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.604206 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.604286 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:38.604467 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.605592 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.606969 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.607637 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.607693 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.607742 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.608087 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.608136 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
I20260926 01:05:38.608850 10878 tablet_service.cc:1467] Processing CreateTablet for tablet 85a94bca113142299926f5c549c16f57 (DEFAULT_TABLE table=test-table [id=45be1874f38a4a4f9360d2a823728c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:38.608832 11012 tablet_service.cc:1467] Processing CreateTablet for tablet 85a94bca113142299926f5c549c16f57 (DEFAULT_TABLE table=test-table [id=45be1874f38a4a4f9360d2a823728c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:38.609081 10744 tablet_service.cc:1467] Processing CreateTablet for tablet 85a94bca113142299926f5c549c16f57 (DEFAULT_TABLE table=test-table [id=45be1874f38a4a4f9360d2a823728c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:05:38.609207 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a94bca113142299926f5c549c16f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:38.609207 11012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a94bca113142299926f5c549c16f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:38.609388 10744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a94bca113142299926f5c549c16f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:05:38.612787 11097 tablet_bootstrap.cc:492] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140: Bootstrap starting.
I20260926 01:05:38.612848 11096 tablet_bootstrap.cc:492] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: Bootstrap starting.
I20260926 01:05:38.613824 11097 tablet_bootstrap.cc:654] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:38.613824 11096 tablet_bootstrap.cc:654] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:38.614544 11095 tablet_bootstrap.cc:492] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: Bootstrap starting.
I20260926 01:05:38.614555 11097 log.cc:824] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:38.614573 11096 log.cc:824] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:38.615236 11095 tablet_bootstrap.cc:654] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: Neither blocks nor log segments found. Creating new log.
I20260926 01:05:38.615484 11097 tablet_bootstrap.cc:492] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140: No bootstrap required, opened a new log
I20260926 01:05:38.615612 11097 ts_tablet_manager.cc:1397] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:05:38.615636 11096 tablet_bootstrap.cc:492] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: No bootstrap required, opened a new log
I20260926 01:05:38.615707 11096 ts_tablet_manager.cc:1397] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 01:05:38.615808 11095 log.cc:824] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:05:38.616614 11095 tablet_bootstrap.cc:492] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: No bootstrap required, opened a new log
I20260926 01:05:38.616688 11095 ts_tablet_manager.cc:1397] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 01:05:38.617488 11097 raft_consensus.cc:348] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd15551d42441088e7cec7cfc16d140" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40999 } } peers { permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 } }
I20260926 01:05:38.617640 11097 raft_consensus.cc:374] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:38.617672 11097 raft_consensus.cc:729] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd15551d42441088e7cec7cfc16d140, State: Initialized, Role: FOLLOWER
I20260926 01:05:38.617594 11096 raft_consensus.cc:348] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd15551d42441088e7cec7cfc16d140" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40999 } } peers { permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 } }
I20260926 01:05:38.617805 11096 raft_consensus.cc:374] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:38.617835 11097 consensus_queue.cc:260] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd15551d42441088e7cec7cfc16d140" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40999 } } peers { permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 } }
I20260926 01:05:38.617856 11096 raft_consensus.cc:729] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bb93b24d944765b4bcfe6780550fff, State: Initialized, Role: FOLLOWER
I20260926 01:05:38.618132 11097 ts_tablet_manager.cc:1428] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:05:38.618108 11096 consensus_queue.cc:260] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd15551d42441088e7cec7cfc16d140" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40999 } } peers { permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 } }
I20260926 01:05:38.618188 10810 heartbeater.cc:503] Master 127.0.98.62:43325 was elected leader, sending a full tablet report...
I20260926 01:05:38.618366 11096 ts_tablet_manager.cc:1428] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:05:38.618592 10944 heartbeater.cc:503] Master 127.0.98.62:43325 was elected leader, sending a full tablet report...
I20260926 01:05:38.618759 11095 raft_consensus.cc:348] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd15551d42441088e7cec7cfc16d140" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40999 } } peers { permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 } }
I20260926 01:05:38.618911 11095 raft_consensus.cc:374] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:05:38.618992 11095 raft_consensus.cc:729] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb79d784ebc4508a430915df2ffb29e, State: Initialized, Role: FOLLOWER
I20260926 01:05:38.619143 11095 consensus_queue.cc:260] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd15551d42441088e7cec7cfc16d140" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40999 } } peers { permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 } }
I20260926 01:05:38.619483 11095 ts_tablet_manager.cc:1428] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:05:38.619498 11078 heartbeater.cc:503] Master 127.0.98.62:43325 was elected leader, sending a full tablet report...
W20260926 01:05:38.670650 10945 tablet.cc:2369] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:05:38.728621 11101 raft_consensus.cc:482] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:05:38.728754 11101 raft_consensus.cc:504] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd15551d42441088e7cec7cfc16d140" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40999 } } peers { permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 } }
I20260926 01:05:38.729097 11101 leader_election.cc:288] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb79d784ebc4508a430915df2ffb29e (127.0.98.3:44159), f5bb93b24d944765b4bcfe6780550fff (127.0.98.2:46329)
I20260926 01:05:38.731884 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.729382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48171 (local address 127.0.98.2:46329)
0926 01:05:38.729422 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.729424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.729434 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:05:38.729492 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.729492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.729502 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.729555 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:38.729692 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.730164 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.731163 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.731548 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.731610 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.731622 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.731763 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.731799 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:05:38.731913 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.729288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46329 (local address 127.0.98.1:48171)
0926 01:05:38.729342 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:38.729356 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:38.729423 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:38.729560 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:38.729560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:38.729561 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:38.729630 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.729632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.730242 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.730243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.731067 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.731069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.731626 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.731627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.731700 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.731703 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:38.731750 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:05:38.731799 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":27}
I20260926 01:05:38.731935 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.729412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46463 (local address 127.0.98.3:44159)
0926 01:05:38.729453 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.729455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.729549 (+ 94us) server_negotiation.cc:407] Connection header received
0926 01:05:38.729595 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.729595 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.729606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.729655 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:05:38.729849 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.730309 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.731255 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.731620 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.731679 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.731690 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.731830 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.731867 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:05:38.731995 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.729288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44159 (local address 127.0.98.1:46463)
0926 01:05:38.729477 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:38.729485 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:05:38.729545 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:05:38.729667 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:05:38.729668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:05:38.729670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:05:38.729776 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.729778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.730405 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.730406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.731163 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:05:38.731165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.731699 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.731700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:05:38.731764 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.731767 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:05:38.731813 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:05:38.731854 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":63,"thread_start_us":76,"threads_started":1}
I20260926 01:05:38.732270 10898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a94bca113142299926f5c549c16f57" candidate_uuid: "3dd15551d42441088e7cec7cfc16d140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bb93b24d944765b4bcfe6780550fff" is_pre_election: true
I20260926 01:05:38.732384 11032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a94bca113142299926f5c549c16f57" candidate_uuid: "3dd15551d42441088e7cec7cfc16d140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb79d784ebc4508a430915df2ffb29e" is_pre_election: true
I20260926 01:05:38.732514 10898 raft_consensus.cc:2454] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd15551d42441088e7cec7cfc16d140 in term 0.
I20260926 01:05:38.732546 11032 raft_consensus.cc:2454] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd15551d42441088e7cec7cfc16d140 in term 0.
I20260926 01:05:38.732805 10697 leader_election.cc:302] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb79d784ebc4508a430915df2ffb29e, 3dd15551d42441088e7cec7cfc16d140; no voters:
I20260926 01:05:38.732947 11101 raft_consensus.cc:2781] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:05:38.733006 11101 raft_consensus.cc:482] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:05:38.733024 11101 raft_consensus.cc:3037] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:38.733862 11101 raft_consensus.cc:504] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd15551d42441088e7cec7cfc16d140" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40999 } } peers { permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 } }
I20260926 01:05:38.733997 11101 leader_election.cc:288] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb79d784ebc4508a430915df2ffb29e (127.0.98.3:44159), f5bb93b24d944765b4bcfe6780550fff (127.0.98.2:46329)
I20260926 01:05:38.734206 11032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a94bca113142299926f5c549c16f57" candidate_uuid: "3dd15551d42441088e7cec7cfc16d140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb79d784ebc4508a430915df2ffb29e"
I20260926 01:05:38.734314 11032 raft_consensus.cc:3037] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:38.734344 10898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a94bca113142299926f5c549c16f57" candidate_uuid: "3dd15551d42441088e7cec7cfc16d140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bb93b24d944765b4bcfe6780550fff"
I20260926 01:05:38.734469 10898 raft_consensus.cc:3037] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:05:38.735128 11032 raft_consensus.cc:2454] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd15551d42441088e7cec7cfc16d140 in term 1.
I20260926 01:05:38.735248 10898 raft_consensus.cc:2454] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd15551d42441088e7cec7cfc16d140 in term 1.
I20260926 01:05:38.735330 10697 leader_election.cc:302] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb79d784ebc4508a430915df2ffb29e, 3dd15551d42441088e7cec7cfc16d140; no voters:
I20260926 01:05:38.735453 11101 raft_consensus.cc:2781] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:05:38.735618 11101 raft_consensus.cc:686] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd15551d42441088e7cec7cfc16d140, State: Running, Role: LEADER
I20260926 01:05:38.735714 11101 consensus_queue.cc:237] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd15551d42441088e7cec7cfc16d140" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40999 } } peers { permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } } peers { permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 } }
I20260926 01:05:38.736891 10619 catalog_manager.cc:5495] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dd15551d42441088e7cec7cfc16d140 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3dd15551d42441088e7cec7cfc16d140" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd15551d42441088e7cec7cfc16d140" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 } health_report { overall_health: UNKNOWN } } }
W20260926 01:05:38.752949 10811 tablet.cc:2369] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:05:38.754842 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.751803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.0.98.1:40999)
0926 01:05:38.751850 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:05:38.751851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:05:38.751859 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:05:38.751907 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:05:38.751908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:05:38.751928 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:05:38.751987 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:05:38.752158 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.752872 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.753891 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:05:38.754149 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:05:38.754206 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:05:38.754439 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:05:38.754587 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:05:38.754673 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:05:38.754696 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 01:05:38.754730 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:05:38.757397 11032 raft_consensus.cc:1261] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e [term 1 FOLLOWER]: Refusing update from remote peer 3dd15551d42441088e7cec7cfc16d140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:38.757396 10898 raft_consensus.cc:1261] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff [term 1 FOLLOWER]: Refusing update from remote peer 3dd15551d42441088e7cec7cfc16d140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:05:38.757735 11105 consensus_queue.cc:1035] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb79d784ebc4508a430915df2ffb29e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:38.757869 11101 consensus_queue.cc:1035] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bb93b24d944765b4bcfe6780550fff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:05:38.760843 11110 mvcc.cc:205] Tried to move back new op lower bound from 7333415889947828224 to 7333415889862270976. Current Snapshot: MvccSnapshot[applied={T|T < 7333415889947828224}]
I20260926 01:05:38.761341 11108 mvcc.cc:205] Tried to move back new op lower bound from 7333415889947828224 to 7333415889862270976. Current Snapshot: MvccSnapshot[applied={T|T < 7333415889947828224 or (T in {7333415889947828224})}]
I20260926 01:05:38.762734 10619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:38.762833 10619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:05:38.763571 10619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 186a57cf595244bbbf82cdf50da0c9e1: Sending DeleteTablet for 3 replicas of tablet 85a94bca113142299926f5c549c16f57
I20260926 01:05:38.764019 10744 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a94bca113142299926f5c549c16f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59444
I20260926 01:05:38.764122 10878 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a94bca113142299926f5c549c16f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35140
I20260926 01:05:38.764144 11012 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a94bca113142299926f5c549c16f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:05:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46706
I20260926 01:05:38.764397 11118 tablet_replica.cc:330] T 85a94bca113142299926f5c549c16f57 P 3dd15551d42441088e7cec7cfc16d140: stopping tablet replica
I20260926 01:05:38.764525 11119 tablet_replica.cc:330] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: stopping tablet replica
I20260926 01:05:38.764633 11119 raft_consensus.cc:2229] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:38.764644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 10681
I20260926 01:05:38.764755 11119 raft_consensus.cc:2258] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:38.765654 11119 ts_tablet_manager.cc:1905] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:38.767185 11119 ts_tablet_manager.cc:1918] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:38.767261 11119 log.cc:1196] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/85a94bca113142299926f5c549c16f57
I20260926 01:05:38.767472 11119 ts_tablet_manager.cc:1939] T 85a94bca113142299926f5c549c16f57 P f5bb93b24d944765b4bcfe6780550fff: Deleting consensus metadata
I20260926 01:05:38.767855 10606 catalog_manager.cc:4841] TS f5bb93b24d944765b4bcfe6780550fff (127.0.98.2:46329): tablet 85a94bca113142299926f5c549c16f57 (table test-table [id=45be1874f38a4a4f9360d2a823728c2c]) successfully deleted
W20260926 01:05:38.770102 10603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40999 (error 108)
I20260926 01:05:38.770305 11120 tablet_replica.cc:330] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: stopping tablet replica
I20260926 01:05:38.770474 11120 raft_consensus.cc:2229] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:05:38.770624 11120 raft_consensus.cc:2258] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:05:38.770655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 10815
I20260926 01:05:38.770660 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:05:38.770537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40999 (local address 127.0.0.1:59468)
0926 01:05:38.770589 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 01:05:38.770628 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:05:38.771492 11120 ts_tablet_manager.cc:1905] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:05:38.773037 11120 ts_tablet_manager.cc:1918] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:05:38.773102 11120 log.cc:1196] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/85a94bca113142299926f5c549c16f57
W20260926 01:05:38.773142 10603 catalog_manager.cc:4584] TS 3dd15551d42441088e7cec7cfc16d140 (127.0.98.1:40999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85a94bca113142299926f5c549c16f57: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40999: connect: Connection refused (error 111)
I20260926 01:05:38.773365 11120 ts_tablet_manager.cc:1939] T 85a94bca113142299926f5c549c16f57 P 1cb79d784ebc4508a430915df2ffb29e: Deleting consensus metadata
I20260926 01:05:38.773772 10603 catalog_manager.cc:4841] TS 1cb79d784ebc4508a430915df2ffb29e (127.0.98.3:44159): tablet 85a94bca113142299926f5c549c16f57 (table test-table [id=45be1874f38a4a4f9360d2a823728c2c]) successfully deleted
I20260926 01:05:38.776065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 10949
I20260926 01:05:38.780333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 10587
2026-09-26T01:05:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1924 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[11124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[11124](info): set up 2 sockets
Sep 26 01:05:38 dist-test-slave-l73v krb5kdc[11124](info): commencing operation
krb5kdc: starting...
W20260926 01:05:39.743096 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:05:39 dist-test-slave-l73v krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:05:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 01:05:39.908527 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasky1EZ9q/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/postgres -l logfile start
2026-09-26 01:05:42.584 UTC [11154] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 01:05:42.584 UTC [11154] LOG: listening on IPv4 address "127.0.98.20", port 38751
2026-09-26 01:05:42.588 UTC [11154] LOG: listening on Unix socket "/tmp/.s.PGSQL.38751"
2026-09-26 01:05:42.615 UTC [11157] LOG: database system was shut down at 2026-09-26 01:05:40 UTC
2026-09-26 01:05:42.619 UTC [11154] LOG: database system is ready to accept connections
I20260926 01:05:43.491039 392 mini_postgres.cc:96] Postgres bound to 38751
2026-09-26 01:05:43.497 UTC [11165] FATAL: database "slave" does not exist
127.0.98.20:38751 - accepting connections
I20260926 01:05:43.497625 392 mini_ranger.cc:161] Starting Ranger...
I20260926 01:05:43.513676 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 01:05:44.051236 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 01:05:44.051321 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-admin
2026-09-26 01:05:44,352 [I] DB FLAVOR :POSTGRES
2026-09-26 01:05:44,353 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:05:44,353 [I] Checking connection..
2026-09-26 01:05:44,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 01:05:44,616 [I] Checking connection passed.
2026-09-26 01:05:44,617 [I] --------- Verifying version history table ---------
2026-09-26 01:05:44,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:05:44,875 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:05:44,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:05:45,131 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:05:45,131 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 01:05:45,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 01:05:45.383 UTC [11229] WARNING: there is no transaction in progress
2026-09-26 01:05:45,389 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 01:05:45,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:05:45,633 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 01:05:45,633 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:05:45,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 01:05:45,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 01:05:46,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-l73v.c.gcp-upstream.internal','N') ;"
2026-09-26 01:05:46,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:05:46,627 [I] Table x_portal_user does not exist in database ranger
2026-09-26 01:05:46,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:05:46,882 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 01:05:46,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:05:47,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:05:47,365 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 01:05:47,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 01:05:48.439 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.474 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.494 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.515 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.535 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.555 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.621 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.630 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.647 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.659 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.672 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.689 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.701 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.710 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.718 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:48.725 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 01:05:49,497 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 01:05:49,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 01:05:49,739 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 01:05:49,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:05:49,993 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 01:05:49,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:05:50,259 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 01:05:50,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:05:50,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:05:50,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-l73v.c.gcp-upstream.internal';"
2026-09-26 01:05:51,063 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 01:05:51,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 01:05:51,311 [I] DB_PATCHES have already been applied
I20260926 01:05:51.316711 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-admin:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasky1EZ9q/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 01:05:51.316782 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 01:05:51.319842 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:48421
Sep 26, 2026 1:05:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:05:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:05:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:05:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:05:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:05:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:05:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 01:05:52.329957 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.832s user 0.000s sys 0.009s
Sep 26, 2026 1:06:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:06:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:06:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:06:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:06:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 01:06:14.993381 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 01:06:15 dist-test-slave-l73v krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384775, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 01:06:15.080593 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 01:06:15.090723 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 01:06:16.096540 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 01:06:16.096623 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-kms
I20260926 01:06:16.096640 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:38751
2026-09-26 01:06:16,268 [I] DB FLAVOR :POSTGRES
2026-09-26 01:06:16,268 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:06:16,268 [I] Checking connection
2026-09-26 01:06:16,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 01:06:16,525 [I] connection success
2026-09-26 01:06:16,525 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 01:06:16,525 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 01:06:16,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 01:06:16,796 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 01:06:16,796 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:06:16,797 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 01:06:16,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38751/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 01:06:17,072 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 01:06:17.596405 392 mini_ranger_kms.cc:310] Created kms service
I20260926 01:06:17.831990 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 01:06:17.877673 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 01:06:17 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384775, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:06:18.353586 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 01:06:18.353700 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-kms:/tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 01:06:18.353719 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 01:06:18.359521 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11518
I20260926 01:06:18.359629 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:58261
Sep 26, 2026 1:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 01:06:20.457862 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.377s user 0.000s sys 0.154s
I20260926 01:06:20.457960 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 01:06:20.458029 392 mini_ranger_kms.cc:401] 127.0.98.20:58261/kms/v1/keys
Sep 26 01:06:20 dist-test-slave-l73v krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790384780, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 01:06:21 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384780, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 01:06:21 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384780, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 01:06:24 dist-test-slave-l73v krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:06:24.085844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44949
--ranger_config_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58261/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:06:24.192440 11568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:06:24.192694 11568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:06:24.192736 11568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:06:24.193982 11568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:06:24.194034 11568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:06:24.194053 11568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:06:24.194069 11568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:06:24.194084 11568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:06:24.195793 11568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:58261/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44949
--ranger_config_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:06:24.196051 11568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:06:24.196347 11568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:06:24.201409 11574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:06:24.201411 11573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:06:24.201411 11576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:06:24.201831 11568 server_base.cc:1034] running on GCE node
I20260926 01:06:24.202102 11568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:06:24.202406 11568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:06:24.203588 11568 hybrid_clock.cc:648] HybridClock initialized: now 1790384784203567 us; error 32 us; skew 500 ppm
Sep 26 01:06:24 dist-test-slave-l73v krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:06:24.212455 11568 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:06:24.212985 11568 webserver.cc:460] Webserver started at http://127.0.98.62:42127/ using document root <none> and password file <none>
I20260926 01:06:24.213202 11568 fs_manager.cc:359] Metadata directory not provided
I20260926 01:06:24.213244 11568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:06:24.213377 11568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:06:24 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:06:24.271224 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "11aeb85029c14ea988c982965c17abcb"
format_stamp: "Formatted at 2026-09-26 01:06:24 on dist-test-slave-l73v"
server_key: "50d8ffdb0ee0dbd7529f74c74dc30477"
server_key_iv: "5fe51917c0e3b2be40a0c302f728f025"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:24.271719 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "11aeb85029c14ea988c982965c17abcb"
format_stamp: "Formatted at 2026-09-26 01:06:24 on dist-test-slave-l73v"
server_key: "50d8ffdb0ee0dbd7529f74c74dc30477"
server_key_iv: "5fe51917c0e3b2be40a0c302f728f025"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:24.273584 11568 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 01:06:24.286460 11583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:06:24.286824 11568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:06:24.286933 11568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "11aeb85029c14ea988c982965c17abcb"
format_stamp: "Formatted at 2026-09-26 01:06:24 on dist-test-slave-l73v"
server_key: "50d8ffdb0ee0dbd7529f74c74dc30477"
server_key_iv: "5fe51917c0e3b2be40a0c302f728f025"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:24.287036 11568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:06:24.297776 11568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:06:24.299777 11568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:06:24.299919 11568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:06:24.305052 11568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44949
I20260926 01:06:24.305090 11635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44949 every 8 connection(s)
I20260926 01:06:24.305545 11568 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:06:24.306279 11636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:06:24.308588 11636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb: Bootstrap starting.
I20260926 01:06:24.309170 11636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb: Neither blocks nor log segments found. Creating new log.
I20260926 01:06:24.309412 11636 log.cc:824] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:06:24.310137 11636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb: No bootstrap required, opened a new log
I20260926 01:06:24.312187 11636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aeb85029c14ea988c982965c17abcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260926 01:06:24.312331 11636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:06:24.312363 11636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11aeb85029c14ea988c982965c17abcb, State: Initialized, Role: FOLLOWER
I20260926 01:06:24.312505 11636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aeb85029c14ea988c982965c17abcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260926 01:06:24.312578 11636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:06:24.312618 11636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:06:24.312664 11636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:06:24.313022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 11568
I20260926 01:06:24.313120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:06:24.313225 11636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aeb85029c14ea988c982965c17abcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260926 01:06:24.313311 392 external_mini_cluster.cc:1428] Setting key 7af2d5f124caf1fd78b55eed67e92e5d
I20260926 01:06:24.313303 11636 leader_election.cc:302] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11aeb85029c14ea988c982965c17abcb; no voters:
I20260926 01:06:24.313444 11636 leader_election.cc:288] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:06:24.313691 11636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:06:24.314244 11639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:06:24.314457 11639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [term 1 LEADER]: Becoming Leader. State: Replica: 11aeb85029c14ea988c982965c17abcb, State: Running, Role: LEADER
I20260926 01:06:24.315225 11639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aeb85029c14ea988c982965c17abcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260926 01:06:24.315577 11639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11aeb85029c14ea988c982965c17abcb. Latest consensus state: current_term: 1 leader_uuid: "11aeb85029c14ea988c982965c17abcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aeb85029c14ea988c982965c17abcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } } }
I20260926 01:06:24.315748 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11aeb85029c14ea988c982965c17abcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11aeb85029c14ea988c982965c17abcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } } }
I20260926 01:06:24.316233 11636 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 01:06:24.316282 11639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [sys.catalog]: This master's current role is: LEADER
I20260926 01:06:24.316813 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb [sys.catalog]: This master's current role is: LEADER
I20260926 01:06:24.319562 11655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:06:24.320885 11655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 01:06:24 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:06:24.326923 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:24.313886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.0.98.62:44949)
0926 01:06:24.314068 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:24.314079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:24.314124 (+ 45us) server_negotiation.cc:407] Connection header received
0926 01:06:24.314291 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:24.314298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:24.314353 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:24.314608 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:06:24.315373 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:24.316890 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:24.318433 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:24.320184 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:24.320385 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:24.323159 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:06:24.323163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:06:24.323164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:06:24.323176 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:06:24.325195 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:06:24.325610 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:06:24.325613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:06:24.325614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:06:24.325714 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:06:24.325908 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:06:24.325909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:06:24.325909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:06:24.326244 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:06:24.326447 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:24.326553 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 01:06:24.326693 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260926 01:06:24.328861 11655 catalog_manager.cc:1347] Generated new cluster ID: 34ad2b62f31f463e8091c017fd512b96
I20260926 01:06:24.328917 11655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:06:24.344017 11655 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:06:24.344869 11655 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:06:24.354092 11655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb: Generated new TSK 0
I20260926 01:06:24.354489 11655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:06:25.108822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44949
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:06:25.236300 11681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:06:25.236599 11681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:06:25.236635 11681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:06:25.238644 11681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:06:25.238730 11681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:06:25.238775 11681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:06:25.241489 11681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:58261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:06:25.241798 11681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:06:25.242163 11681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:06:25.249025 11686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:06:25.249238 11687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:06:25.249437 11689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:06:25.249552 11681 server_base.cc:1034] running on GCE node
I20260926 01:06:25.249805 11681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:06:25.250156 11681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:06:25.251341 11681 hybrid_clock.cc:648] HybridClock initialized: now 1790384785251314 us; error 44 us; skew 500 ppm
Sep 26 01:06:25 dist-test-slave-l73v krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:06:25.254824 11681 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:06:25.255524 11681 webserver.cc:460] Webserver started at http://127.0.98.1:46311/ using document root <none> and password file <none>
I20260926 01:06:25.255838 11681 fs_manager.cc:359] Metadata directory not provided
I20260926 01:06:25.255923 11681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:06:25.256093 11681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:06:25 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:06:25.271468 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "8d2864066351462bbb5d38669ca86c0e"
format_stamp: "Formatted at 2026-09-26 01:06:25 on dist-test-slave-l73v"
server_key: "b228ad5dbd0d0da73612e6bee2fc37e9"
server_key_iv: "5e5d20bcb2e2ed2860068b843f7827ff"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:25.271982 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8d2864066351462bbb5d38669ca86c0e"
format_stamp: "Formatted at 2026-09-26 01:06:25 on dist-test-slave-l73v"
server_key: "b228ad5dbd0d0da73612e6bee2fc37e9"
server_key_iv: "5e5d20bcb2e2ed2860068b843f7827ff"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:25.273973 11681 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 01:06:25.289944 11696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:06:25.290140 11681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 01:06:25.290228 11681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "8d2864066351462bbb5d38669ca86c0e"
format_stamp: "Formatted at 2026-09-26 01:06:25 on dist-test-slave-l73v"
server_key: "b228ad5dbd0d0da73612e6bee2fc37e9"
server_key_iv: "5e5d20bcb2e2ed2860068b843f7827ff"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:25.290330 11681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:06:25.315261 11681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:06:25.318077 11681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:06:25.318280 11681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:06:25.318958 11681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:06:25.319737 11681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:06:25.319801 11681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:06:25.319841 11681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:06:25.319867 11681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:06:25.327555 11681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33923
I20260926 01:06:25.328042 11681 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:06:25.329612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 11681
I20260926 01:06:25.329701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:06:25.329883 392 external_mini_cluster.cc:1428] Setting key 980287779727278d1c38cc94c8d61dc3
I20260926 01:06:25.338792 11811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33923 every 8 connection(s)
Sep 26 01:06:25 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 01:06:25.351720 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.333751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44949 (local address 127.0.98.1:52575)
0926 01:06:25.334039 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 01:06:25.334072 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 01:06:25.334742 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:06:25.335154 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:06:25.335160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:06:25.335429 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:06:25.335719 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:25.335726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.336678 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.336679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:25.337865 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:25.337869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.338462 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.338463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:25.338635 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:25.339279 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:06:25.339290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:06:25.340947 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:06:25.348150 (+ 7203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:06:25.348165 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:06:25.348168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:06:25.348531 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:06:25.350366 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:06:25.350368 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:06:25.350368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:06:25.350615 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:06:25.350955 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:06:25.350965 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:06:25.351174 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 01:06:25.351280 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 01:06:25.351474 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
I20260926 01:06:25.352183 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.333757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52575 (local address 127.0.98.62:44949)
0926 01:06:25.333908 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:25.333910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:25.334890 (+ 980us) server_negotiation.cc:407] Connection header received
0926 01:06:25.334922 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:25.334923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:25.334940 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:25.335012 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:06:25.335829 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.336579 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.337973 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.338272 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.338338 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:25.341076 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:06:25.341078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:06:25.341078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:06:25.341087 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:06:25.347815 (+ 6728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:06:25.348626 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:06:25.348628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:06:25.348628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:06:25.348702 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:06:25.350704 (+ 2002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:06:25.350706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:06:25.350706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:06:25.350869 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:06:25.350929 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:25.351992 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0926 01:06:25.352089 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260926 01:06:25.352656 11812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44949
I20260926 01:06:25.352763 11812 heartbeater.cc:464] Registering TS with master...
I20260926 01:06:25.352984 11812 heartbeater.cc:511] Master 127.0.98.62:44949 requested a full tablet report, sending...
I20260926 01:06:25.353515 11600 ts_manager.cc:194] Registered new tserver with Master: 8d2864066351462bbb5d38669ca86c0e (127.0.98.1:33923)
I20260926 01:06:25.354264 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:06:25.369880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44949
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:06:25.495659 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:06:25.496054 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:06:25.496192 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:06:25.498237 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:06:25.498629 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:06:25.498740 11818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:06:25.501451 11818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:58261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:06:25.501996 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:06:25.502413 11818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:06:25.511618 11826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:06:25.511971 11828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:06:25.512406 11825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:06:25.513796 11818 server_base.cc:1034] running on GCE node
I20260926 01:06:25.514035 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:06:25.514317 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:06:25.515663 11818 hybrid_clock.cc:648] HybridClock initialized: now 1790384785515635 us; error 50 us; skew 500 ppm
Sep 26 01:06:25 dist-test-slave-l73v krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:06:25.519358 11818 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:06:25.519986 11818 webserver.cc:460] Webserver started at http://127.0.98.2:32969/ using document root <none> and password file <none>
I20260926 01:06:25.520246 11818 fs_manager.cc:359] Metadata directory not provided
I20260926 01:06:25.520304 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:06:25.520437 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:06:25 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:06:25.541500 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "61f6f9a2ec9c47b9a5115505a31a986e"
format_stamp: "Formatted at 2026-09-26 01:06:25 on dist-test-slave-l73v"
server_key: "63e8434941adc57296cde6401716ab6c"
server_key_iv: "9264ca2ace5c55ba1fb078aa06e6141a"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:25.542061 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "61f6f9a2ec9c47b9a5115505a31a986e"
format_stamp: "Formatted at 2026-09-26 01:06:25 on dist-test-slave-l73v"
server_key: "63e8434941adc57296cde6401716ab6c"
server_key_iv: "9264ca2ace5c55ba1fb078aa06e6141a"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:25.546810 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
Sep 26 01:06:25 dist-test-slave-l73v krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:06:25.570731 11835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:06:25.570990 11818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 01:06:25.571105 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "61f6f9a2ec9c47b9a5115505a31a986e"
format_stamp: "Formatted at 2026-09-26 01:06:25 on dist-test-slave-l73v"
server_key: "63e8434941adc57296cde6401716ab6c"
server_key_iv: "9264ca2ace5c55ba1fb078aa06e6141a"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:25.571233 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:06:25.628420 11818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:06:25.631472 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:06:25.631624 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:06:25.631835 11818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:06:25.632227 11818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:06:25.632269 11818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:06:25.632300 11818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:06:25.632315 11818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:06:25.640655 11818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46443
I20260926 01:06:25.641098 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:06:25.642912 11949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46443 every 8 connection(s)
I20260926 01:06:25.643476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 11818
I20260926 01:06:25.643573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:06:25.643760 392 external_mini_cluster.cc:1428] Setting key 49c269636b87ef58bce7cc6a3d3c8146
Sep 26 01:06:25 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:06:25.655107 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.643853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40451 (local address 127.0.98.62:44949)
0926 01:06:25.643899 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:25.643901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:25.644765 (+ 864us) server_negotiation.cc:407] Connection header received
0926 01:06:25.645440 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:25.645442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:25.645453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:25.645481 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:06:25.646343 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.647091 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.648688 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.649011 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.649053 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:25.651683 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:06:25.651684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:06:25.651685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:06:25.651693 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:06:25.653344 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:06:25.653886 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:06:25.653887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:06:25.653888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:06:25.653953 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:06:25.654257 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:06:25.654258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:06:25.654258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:06:25.654445 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:06:25.654488 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:25.654896 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 01:06:25.654999 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:06:25.655705 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.643551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44949 (local address 127.0.98.2:40451)
0926 01:06:25.644646 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0926 01:06:25.644690 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:06:25.645308 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:06:25.645689 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:06:25.645693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:06:25.645937 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:06:25.646228 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:25.646236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.647237 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.647238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:25.648579 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:25.648582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.649147 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.649147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:25.649287 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:25.649916 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:06:25.649926 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:06:25.651560 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:06:25.653458 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:06:25.653467 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:06:25.653470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:06:25.653789 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:06:25.654022 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:06:25.654023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:06:25.654023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:06:25.654173 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:06:25.654533 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:06:25.654540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:06:25.654747 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 01:06:25.655252 (+ 505us) client_negotiation.cc:240] Negotiation successful
0926 01:06:25.655460 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"thread_start_us":104,"threads_started":1}
I20260926 01:06:25.656198 11950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44949
I20260926 01:06:25.656299 11950 heartbeater.cc:464] Registering TS with master...
I20260926 01:06:25.656512 11950 heartbeater.cc:511] Master 127.0.98.62:44949 requested a full tablet report, sending...
I20260926 01:06:25.656896 11600 ts_manager.cc:194] Registered new tserver with Master: 61f6f9a2ec9c47b9a5115505a31a986e (127.0.98.2:46443)
I20260926 01:06:25.657406 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40451
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:06:25.691720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44949
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:06:25.831717 11955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:06:25.832173 11955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:06:25.832273 11955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:06:25.834254 11955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:06:25.834373 11955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:06:25.834527 11955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:06:25.837080 11955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:58261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:06:25.837646 11955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:06:25.837985 11955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:06:25.844333 11961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:06:25.844333 11960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:06:25.848361 11963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:06:25.849061 11955 server_base.cc:1034] running on GCE node
I20260926 01:06:25.849298 11955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:06:25.849581 11955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:06:25.856980 11955 hybrid_clock.cc:648] HybridClock initialized: now 1790384785856941 us; error 53 us; skew 500 ppm
Sep 26 01:06:25 dist-test-slave-l73v krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:06:25.859798 11955 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:06:25.860275 11955 webserver.cc:460] Webserver started at http://127.0.98.3:45441/ using document root <none> and password file <none>
I20260926 01:06:25.860569 11955 fs_manager.cc:359] Metadata directory not provided
I20260926 01:06:25.860633 11955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:06:25.860761 11955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:06:25 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:06:25.875460 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b008a90021d4dec86095f4782eabbb8"
format_stamp: "Formatted at 2026-09-26 01:06:25 on dist-test-slave-l73v"
server_key: "9e7dde92b793aa561db2b56bdc1afd23"
server_key_iv: "d1dba0316a94699ee52410ae27bf0f2d"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:25.876099 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b008a90021d4dec86095f4782eabbb8"
format_stamp: "Formatted at 2026-09-26 01:06:25 on dist-test-slave-l73v"
server_key: "9e7dde92b793aa561db2b56bdc1afd23"
server_key_iv: "d1dba0316a94699ee52410ae27bf0f2d"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:25.877952 11955 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 01:06:25.907770 11970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:06:25.907999 11955 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260926 01:06:25.908108 11955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "9b008a90021d4dec86095f4782eabbb8"
format_stamp: "Formatted at 2026-09-26 01:06:25 on dist-test-slave-l73v"
server_key: "9e7dde92b793aa561db2b56bdc1afd23"
server_key_iv: "d1dba0316a94699ee52410ae27bf0f2d"
server_key_version: "kuduclusterkey@0"
I20260926 01:06:25.908236 11955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:06:25.924118 11955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:06:25.926752 11955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:06:25.926909 11955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:06:25.927150 11955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:06:25.927515 11955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:06:25.927562 11955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:06:25.927600 11955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:06:25.927628 11955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:06:25.934882 11955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44417
I20260926 01:06:25.935567 11955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:06:25.936070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 11955
I20260926 01:06:25.936178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:06:25.936367 392 external_mini_cluster.cc:1428] Setting key b457f4b89db9807c37989f41f630d709
I20260926 01:06:25.936954 12083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44417 every 8 connection(s)
Sep 26 01:06:25 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:06:25.949851 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.937509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53755 (local address 127.0.98.62:44949)
0926 01:06:25.937554 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:25.937556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:25.937825 (+ 269us) server_negotiation.cc:407] Connection header received
0926 01:06:25.939151 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:25.939153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:25.939166 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:25.939196 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:06:25.940132 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.940856 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.942488 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.942815 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.942863 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:25.945739 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:06:25.945741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:06:25.945741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:06:25.945750 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:06:25.947552 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:06:25.948408 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:06:25.948410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:06:25.948410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:06:25.948480 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:06:25.948836 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:06:25.948837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:06:25.948837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:06:25.948993 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:06:25.949056 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:25.949527 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 01:06:25.949743 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:06:25.950780 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.937509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44949 (local address 127.0.98.3:53755)
0926 01:06:25.937701 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 01:06:25.937747 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 01:06:25.939011 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:06:25.939406 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:06:25.939411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:06:25.939709 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:06:25.940006 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:25.940013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.940988 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.940989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:25.942335 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:25.942338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.942974 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.942975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:25.943147 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:25.943802 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:06:25.943813 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:06:25.945601 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:06:25.947670 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:06:25.947682 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:06:25.947685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:06:25.948316 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:06:25.948556 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:06:25.948557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:06:25.948557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:06:25.948745 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:06:25.949115 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:06:25.949124 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:06:25.949350 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 01:06:25.950356 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0926 01:06:25.950587 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260926 01:06:25.951311 12084 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44949
I20260926 01:06:25.951444 12084 heartbeater.cc:464] Registering TS with master...
I20260926 01:06:25.951655 12084 heartbeater.cc:511] Master 127.0.98.62:44949 requested a full tablet report, sending...
I20260926 01:06:25.952214 11600 ts_manager.cc:194] Registered new tserver with Master: 9b008a90021d4dec86095f4782eabbb8 (127.0.98.3:44417)
I20260926 01:06:25.952721 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53755
I20260926 01:06:25.959043 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:06:25.977270 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.962809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.0.98.62:44949)
0926 01:06:25.962861 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:25.962864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:25.963201 (+ 337us) server_negotiation.cc:407] Connection header received
0926 01:06:25.963929 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:25.963931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:25.963945 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:25.963977 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:06:25.965112 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.965983 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.969733 (+ 3750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.970040 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.970088 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:25.973536 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:06:25.973538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:06:25.973538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:06:25.973548 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:06:25.975339 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:06:25.976204 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:06:25.976206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:06:25.976206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:06:25.976282 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:06:25.976534 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:06:25.976535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:06:25.976535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:06:25.976710 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:06:25.976796 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:25.977046 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 01:06:25.977166 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:06:25.979806 11600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:06:25.982327 11600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:06:25.999794 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.991418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44417 (local address 127.0.0.1:50594)
0926 01:06:25.992803 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0926 01:06:25.992811 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:06:25.992841 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:06:25.993210 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:06:25.993213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:06:25.993216 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:06:25.993298 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:25.993300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.995084 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.995085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:25.998214 (+ 3129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:25.998220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.999459 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.999461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:25.999566 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:25.999585 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 01:06:25.999639 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 01:06:25.999699 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"thread_start_us":72,"threads_started":1}
I20260926 01:06:26.001044 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.991561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.0.98.3:44417)
0926 01:06:25.992972 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:25.992978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:25.992988 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:06:25.993026 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:25.993028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:25.993044 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:25.993107 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:06:25.993371 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.994978 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:25.998361 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:25.999347 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.000762 (+ 1415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:26.000877 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:26.000907 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 01:06:26.000954 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":87,"threads_started":1}
I20260926 01:06:26.001967 12018 tablet_service.cc:1467] Processing CreateTablet for tablet 5a60f05b7435498dafba09b7911aaec7 (DEFAULT_TABLE table=test-table [id=d212e7ac5094442eab23a2ba8931bc10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:06:26.002370 12018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a60f05b7435498dafba09b7911aaec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:06:26.007236 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.991144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46443 (local address 127.0.0.1:55620)
0926 01:06:25.992588 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0926 01:06:25.992640 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:06:25.992705 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:06:26.002720 (+ 10015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:06:26.002721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:06:26.002724 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:06:26.002788 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:26.002791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.004506 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.004507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:26.005775 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:26.005781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.006970 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.006972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:26.007047 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:26.007054 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 01:06:26.007087 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 01:06:26.007140 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"thread_start_us":91,"threads_started":1}
I20260926 01:06:26.007777 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.991693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.0.98.2:46443)
0926 01:06:26.002505 (+ 10812us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:26.002510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:26.002523 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:06:26.002563 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:26.002566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:26.002583 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:26.002867 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:06:26.002949 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.004014 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.006010 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.006832 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.007509 (+ 677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:26.007620 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:26.007651 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 01:06:26.007698 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10763,"thread_start_us":79,"threads_started":1}
I20260926 01:06:26.008610 11884 tablet_service.cc:1467] Processing CreateTablet for tablet 5a60f05b7435498dafba09b7911aaec7 (DEFAULT_TABLE table=test-table [id=d212e7ac5094442eab23a2ba8931bc10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:06:26.009011 11884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a60f05b7435498dafba09b7911aaec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:06:26.009011 12103 tablet_bootstrap.cc:492] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: Bootstrap starting.
I20260926 01:06:26.010663 12103 tablet_bootstrap.cc:654] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: Neither blocks nor log segments found. Creating new log.
I20260926 01:06:26.018450 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.991991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33923 (local address 127.0.0.1:36568)
0926 01:06:26.000634 (+ 8643us) negotiation.cc:107] Waiting for socket to connect
0926 01:06:26.000645 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:06:26.000684 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:06:26.007775 (+ 7091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:06:26.007776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:06:26.007779 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:06:26.007823 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:26.007825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.015336 (+ 7511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.015338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:26.016884 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:26.016888 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.018113 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.018115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:26.018185 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:26.018189 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:06:26.018235 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:06:26.018292 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8628,"thread_start_us":60,"threads_started":1}
I20260926 01:06:26.018795 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:25.992211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.0.98.1:33923)
0926 01:06:26.004239 (+ 12028us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:26.004244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:26.004255 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:06:26.004290 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:26.004292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:26.004309 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:26.004394 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:06:26.013772 (+ 9378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.015217 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.017059 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.017920 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.017998 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:26.018064 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:26.018667 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 01:06:26.018722 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11985,"thread_start_us":50,"threads_started":1}
I20260926 01:06:26.019248 12103 log.cc:824] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: Log is configured to *not* fsync() on all Append() calls
I20260926 01:06:26.019503 11728 tablet_service.cc:1467] Processing CreateTablet for tablet 5a60f05b7435498dafba09b7911aaec7 (DEFAULT_TABLE table=test-table [id=d212e7ac5094442eab23a2ba8931bc10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:06:26.019768 12104 tablet_bootstrap.cc:492] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: Bootstrap starting.
I20260926 01:06:26.020016 11728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a60f05b7435498dafba09b7911aaec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:06:26.021023 12104 tablet_bootstrap.cc:654] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: Neither blocks nor log segments found. Creating new log.
I20260926 01:06:26.021416 12104 log.cc:824] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:06:26.022454 12103 tablet_bootstrap.cc:492] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: No bootstrap required, opened a new log
I20260926 01:06:26.022603 12103 ts_tablet_manager.cc:1397] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260926 01:06:26.024876 12103 raft_consensus.cc:348] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 } } peers { permanent_uuid: "8d2864066351462bbb5d38669ca86c0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33923 } } peers { permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 } }
I20260926 01:06:26.025246 12103 raft_consensus.cc:374] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:06:26.025354 12103 raft_consensus.cc:729] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b008a90021d4dec86095f4782eabbb8, State: Initialized, Role: FOLLOWER
I20260926 01:06:26.025616 12103 consensus_queue.cc:260] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 } } peers { permanent_uuid: "8d2864066351462bbb5d38669ca86c0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33923 } } peers { permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 } }
I20260926 01:06:26.026033 12103 ts_tablet_manager.cc:1428] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:06:26.026198 12084 heartbeater.cc:503] Master 127.0.98.62:44949 was elected leader, sending a full tablet report...
I20260926 01:06:26.028795 12104 tablet_bootstrap.cc:492] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: No bootstrap required, opened a new log
I20260926 01:06:26.028904 12104 ts_tablet_manager.cc:1397] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260926 01:06:26.031803 12104 raft_consensus.cc:348] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 } } peers { permanent_uuid: "8d2864066351462bbb5d38669ca86c0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33923 } } peers { permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 } }
I20260926 01:06:26.032071 12104 raft_consensus.cc:374] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:06:26.032122 12104 raft_consensus.cc:729] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f6f9a2ec9c47b9a5115505a31a986e, State: Initialized, Role: FOLLOWER
I20260926 01:06:26.035240 12104 consensus_queue.cc:260] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 } } peers { permanent_uuid: "8d2864066351462bbb5d38669ca86c0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33923 } } peers { permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 } }
I20260926 01:06:26.035678 12104 ts_tablet_manager.cc:1428] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260926 01:06:26.035840 11950 heartbeater.cc:503] Master 127.0.98.62:44949 was elected leader, sending a full tablet report...
I20260926 01:06:26.036862 12107 tablet_bootstrap.cc:492] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e: Bootstrap starting.
I20260926 01:06:26.037724 12107 tablet_bootstrap.cc:654] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e: Neither blocks nor log segments found. Creating new log.
I20260926 01:06:26.038087 12107 log.cc:824] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:06:26.039860 12107 tablet_bootstrap.cc:492] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e: No bootstrap required, opened a new log
I20260926 01:06:26.039952 12107 ts_tablet_manager.cc:1397] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:06:26.047811 12107 raft_consensus.cc:348] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 } } peers { permanent_uuid: "8d2864066351462bbb5d38669ca86c0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33923 } } peers { permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 } }
I20260926 01:06:26.048086 12107 raft_consensus.cc:374] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:06:26.048125 12107 raft_consensus.cc:729] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2864066351462bbb5d38669ca86c0e, State: Initialized, Role: FOLLOWER
I20260926 01:06:26.048307 12107 consensus_queue.cc:260] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 } } peers { permanent_uuid: "8d2864066351462bbb5d38669ca86c0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33923 } } peers { permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 } }
I20260926 01:06:26.048622 12107 ts_tablet_manager.cc:1428] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260926 01:06:26.048745 11812 heartbeater.cc:503] Master 127.0.98.62:44949 was elected leader, sending a full tablet report...
W20260926 01:06:26.089612 11813 tablet.cc:2369] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 01:06:26 dist-test-slave-l73v krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 01:06:26.144083 11951 tablet.cc:2369] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:06:26.181773 12111 raft_consensus.cc:482] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:06:26.181915 12111 raft_consensus.cc:504] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 } } peers { permanent_uuid: "8d2864066351462bbb5d38669ca86c0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33923 } } peers { permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 } }
I20260926 01:06:26.182276 12111 leader_election.cc:288] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61f6f9a2ec9c47b9a5115505a31a986e (127.0.98.2:46443), 9b008a90021d4dec86095f4782eabbb8 (127.0.98.3:44417)
W20260926 01:06:26.186522 12085 tablet.cc:2369] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:06:26.189694 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:26.182527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33875 (local address 127.0.98.2:46443)
0926 01:06:26.182582 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:26.182584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:26.184078 (+ 1494us) server_negotiation.cc:407] Connection header received
0926 01:06:26.184746 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:26.184747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:26.184762 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:26.184790 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:06:26.185040 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.185719 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.188721 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.189248 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.189295 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:26.189308 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:26.189557 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 01:06:26.189619 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:06:26.189708 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:26.182576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46443 (local address 127.0.98.1:33875)
0926 01:06:26.183999 (+ 1423us) negotiation.cc:107] Waiting for socket to connect
0926 01:06:26.184015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 01:06:26.184648 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:06:26.184861 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:06:26.184862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:06:26.184878 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:06:26.184965 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:26.184967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.185819 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.185820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:26.187532 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:26.187537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.189373 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.189374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:26.189455 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:26.189460 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:06:26.189527 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 01:06:26.189573 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1394,"thread_start_us":92,"threads_started":1}
I20260926 01:06:26.190088 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60f05b7435498dafba09b7911aaec7" candidate_uuid: "8d2864066351462bbb5d38669ca86c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" is_pre_election: true
I20260926 01:06:26.190272 11901 raft_consensus.cc:2454] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2864066351462bbb5d38669ca86c0e in term 0.
I20260926 01:06:26.190626 11698 leader_election.cc:302] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f6f9a2ec9c47b9a5115505a31a986e, 8d2864066351462bbb5d38669ca86c0e; no voters:
I20260926 01:06:26.190837 12111 raft_consensus.cc:2781] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:06:26.190905 12111 raft_consensus.cc:482] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:06:26.190984 12111 raft_consensus.cc:3037] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:06:26.191956 12111 raft_consensus.cc:504] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 } } peers { permanent_uuid: "8d2864066351462bbb5d38669ca86c0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33923 } } peers { permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 } }
I20260926 01:06:26.192106 12111 leader_election.cc:288] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [CANDIDATE]: Term 1 election: Requested vote from peers 61f6f9a2ec9c47b9a5115505a31a986e (127.0.98.2:46443), 9b008a90021d4dec86095f4782eabbb8 (127.0.98.3:44417)
I20260926 01:06:26.192517 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60f05b7435498dafba09b7911aaec7" candidate_uuid: "8d2864066351462bbb5d38669ca86c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f6f9a2ec9c47b9a5115505a31a986e"
I20260926 01:06:26.192611 11901 raft_consensus.cc:3037] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:06:26.193008 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:26.182784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42043 (local address 127.0.98.3:44417)
0926 01:06:26.182825 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:26.182827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:26.189352 (+ 6525us) server_negotiation.cc:407] Connection header received
0926 01:06:26.189384 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:26.189385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:26.189398 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:26.189448 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:06:26.189644 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.190354 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.191949 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.192467 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.192512 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:26.192523 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:26.192895 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 01:06:26.192937 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:06:26.193347 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:26.182727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44417 (local address 127.0.98.1:42043)
0926 01:06:26.189191 (+ 6464us) negotiation.cc:107] Waiting for socket to connect
0926 01:06:26.189206 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 01:06:26.189281 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:06:26.189473 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:06:26.189474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:06:26.189477 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:06:26.189548 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:26.189550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.190507 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.190508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:26.191843 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:06:26.191847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.192737 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.192738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:06:26.192816 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:26.192820 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:06:26.193205 (+ 385us) client_negotiation.cc:240] Negotiation successful
0926 01:06:26.193250 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6437,"thread_start_us":61,"threads_started":1}
I20260926 01:06:26.193495 11901 raft_consensus.cc:2454] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2864066351462bbb5d38669ca86c0e in term 1.
I20260926 01:06:26.194643 11698 leader_election.cc:302] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f6f9a2ec9c47b9a5115505a31a986e, 8d2864066351462bbb5d38669ca86c0e; no voters:
I20260926 01:06:26.194690 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60f05b7435498dafba09b7911aaec7" candidate_uuid: "8d2864066351462bbb5d38669ca86c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b008a90021d4dec86095f4782eabbb8" is_pre_election: true
I20260926 01:06:26.194833 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60f05b7435498dafba09b7911aaec7" candidate_uuid: "8d2864066351462bbb5d38669ca86c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b008a90021d4dec86095f4782eabbb8"
I20260926 01:06:26.194873 12038 raft_consensus.cc:2454] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2864066351462bbb5d38669ca86c0e in term 0.
I20260926 01:06:26.195236 12111 raft_consensus.cc:2781] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:06:26.195392 12111 raft_consensus.cc:686] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [term 1 LEADER]: Becoming Leader. State: Replica: 8d2864066351462bbb5d38669ca86c0e, State: Running, Role: LEADER
I20260926 01:06:26.195465 12111 consensus_queue.cc:237] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 } } peers { permanent_uuid: "8d2864066351462bbb5d38669ca86c0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33923 } } peers { permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 } }
I20260926 01:06:26.196337 11600 catalog_manager.cc:5495] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d2864066351462bbb5d38669ca86c0e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8d2864066351462bbb5d38669ca86c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2864066351462bbb5d38669ca86c0e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 } health_report { overall_health: UNKNOWN } } }
I20260926 01:06:26.233045 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:26.229859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.0.98.1:33923)
0926 01:06:26.229972 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 01:06:26.229973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:06:26.229979 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:06:26.230012 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:06:26.230012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:06:26.230033 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:06:26.230090 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:06:26.230233 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.230939 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.231743 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:06:26.231949 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:06:26.232014 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:06:26.232183 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:06:26.232304 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:06:26.232382 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:06:26.232422 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 01:06:26.232447 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 01:06:26.235334 12037 raft_consensus.cc:3037] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:06:26.236146 12037 raft_consensus.cc:1261] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8 [term 1 FOLLOWER]: Refusing update from remote peer 8d2864066351462bbb5d38669ca86c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:06:26.236455 12114 consensus_queue.cc:1035] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b008a90021d4dec86095f4782eabbb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:06:26.236836 11901 raft_consensus.cc:1261] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e [term 1 FOLLOWER]: Refusing update from remote peer 8d2864066351462bbb5d38669ca86c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:06:26.237268 12114 consensus_queue.cc:1035] T 5a60f05b7435498dafba09b7911aaec7 P 8d2864066351462bbb5d38669ca86c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f6f9a2ec9c47b9a5115505a31a986e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:06:26.239784 12120 mvcc.cc:205] Tried to move back new op lower bound from 7333416084417187840 to 7333416084257038336. Current Snapshot: MvccSnapshot[applied={T|T < 7333416084417187840}]
I20260926 01:06:26.241199 12118 mvcc.cc:205] Tried to move back new op lower bound from 7333416084417187840 to 7333416084257038336. Current Snapshot: MvccSnapshot[applied={T|T < 7333416084417187840}]
I20260926 01:06:26.242339 11600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:06:26.242480 11600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:06:26.243348 11600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11aeb85029c14ea988c982965c17abcb: Sending DeleteTablet for 3 replicas of tablet 5a60f05b7435498dafba09b7911aaec7
I20260926 01:06:26.243693 11884 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a60f05b7435498dafba09b7911aaec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:06:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55620
I20260926 01:06:26.243945 12018 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a60f05b7435498dafba09b7911aaec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:06:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50594
I20260926 01:06:26.244131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 11681
I20260926 01:06:26.244143 11728 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a60f05b7435498dafba09b7911aaec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:06:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36568
I20260926 01:06:26.245417 12126 tablet_replica.cc:330] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: stopping tablet replica
I20260926 01:06:26.245558 12126 raft_consensus.cc:2229] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:06:26.245729 12126 raft_consensus.cc:2258] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:06:26.245904 12127 tablet_replica.cc:330] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: stopping tablet replica
I20260926 01:06:26.246016 12127 raft_consensus.cc:2229] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:06:26.246122 12127 raft_consensus.cc:2258] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:06:26.246629 12127 ts_tablet_manager.cc:1905] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:06:26.246999 12126 ts_tablet_manager.cc:1905] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:06:26.248696 12126 ts_tablet_manager.cc:1918] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:06:26.248852 12126 log.cc:1196] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/5a60f05b7435498dafba09b7911aaec7
I20260926 01:06:26.248840 12127 ts_tablet_manager.cc:1918] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:06:26.248937 12127 log.cc:1196] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/5a60f05b7435498dafba09b7911aaec7
I20260926 01:06:26.249198 12127 ts_tablet_manager.cc:1939] T 5a60f05b7435498dafba09b7911aaec7 P 9b008a90021d4dec86095f4782eabbb8: Deleting consensus metadata
I20260926 01:06:26.249223 12126 ts_tablet_manager.cc:1939] T 5a60f05b7435498dafba09b7911aaec7 P 61f6f9a2ec9c47b9a5115505a31a986e: Deleting consensus metadata
I20260926 01:06:26.249712 11587 catalog_manager.cc:4841] TS 9b008a90021d4dec86095f4782eabbb8 (127.0.98.3:44417): tablet 5a60f05b7435498dafba09b7911aaec7 (table test-table [id=d212e7ac5094442eab23a2ba8931bc10]) successfully deleted
I20260926 01:06:26.249922 11585 catalog_manager.cc:4841] TS 61f6f9a2ec9c47b9a5115505a31a986e (127.0.98.2:46443): tablet 5a60f05b7435498dafba09b7911aaec7 (table test-table [id=d212e7ac5094442eab23a2ba8931bc10]) successfully deleted
W20260926 01:06:26.251617 11584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33923 (error 108)
I20260926 01:06:26.252341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 11818
I20260926 01:06:26.252463 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:06:26.252330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33923 (local address 127.0.0.1:36596)
0926 01:06:26.252386 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:06:26.252424 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 01:06:26.253034 11584 catalog_manager.cc:4584] TS 8d2864066351462bbb5d38669ca86c0e (127.0.98.1:33923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a60f05b7435498dafba09b7911aaec7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33923: connect: Connection refused (error 111)
I20260926 01:06:26.257014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 11955
I20260926 01:06:26.262351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 11568
I20260926 01:06:26.267495 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 01:06:26.602874 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 01:06:26.602989 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 01:06:26.948683 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 01:06:26.948 UTC [11154] LOG: received smart shutdown request
2026-09-26 01:06:26.952 UTC [11154] LOG: background worker "logical replication launcher" (PID 11163) exited with exit code 1
2026-09-26 01:06:26.952 UTC [11158] LOG: shutting down
2026-09-26 01:06:27.180 UTC [11154] LOG: database system is shut down
2026-09-26T01:06:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48481 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:06:27 dist-test-slave-l73v krb5kdc[12137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:06:27 dist-test-slave-l73v krb5kdc[12137](info): set up 2 sockets
Sep 26 01:06:27 dist-test-slave-l73v krb5kdc[12137](info): commencing operation
krb5kdc: starting...
W20260926 01:06:28.230082 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:06:28 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:06:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 01:06:28.402666 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasky1EZ9q/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/postgres -l logfile start
2026-09-26 01:06:31.186 UTC [12167] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 01:06:31.186 UTC [12167] LOG: listening on IPv4 address "127.0.98.20", port 33693
2026-09-26 01:06:31.190 UTC [12167] LOG: listening on Unix socket "/tmp/.s.PGSQL.33693"
2026-09-26 01:06:31.219 UTC [12170] LOG: database system was shut down at 2026-09-26 01:06:29 UTC
2026-09-26 01:06:31.223 UTC [12167] LOG: database system is ready to accept connections
I20260926 01:06:32.082091 392 mini_postgres.cc:96] Postgres bound to 33693
2026-09-26 01:06:32.088 UTC [12178] FATAL: database "slave" does not exist
127.0.98.20:33693 - accepting connections
I20260926 01:06:32.088828 392 mini_ranger.cc:161] Starting Ranger...
I20260926 01:06:32.104121 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 01:06:32.687125 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 01:06:32.687196 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-admin
2026-09-26 01:06:32,976 [I] DB FLAVOR :POSTGRES
2026-09-26 01:06:32,977 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:06:32,977 [I] Checking connection..
2026-09-26 01:06:32,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 01:06:33,230 [I] Checking connection passed.
2026-09-26 01:06:33,230 [I] --------- Verifying version history table ---------
2026-09-26 01:06:33,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:06:33,486 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:06:33,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:06:33,745 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 01:06:33,745 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 01:06:33,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 01:06:34.009 UTC [12242] WARNING: there is no transaction in progress
2026-09-26 01:06:34,015 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 01:06:34,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 01:06:34,257 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 01:06:34,257 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:06:34,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 01:06:34,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 01:06:34,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-l73v.c.gcp-upstream.internal','N') ;"
2026-09-26 01:06:35,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:06:35,323 [I] Table x_portal_user does not exist in database ranger
2026-09-26 01:06:35,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:06:35,566 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 01:06:35,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:06:35,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:06:36,079 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 01:06:36,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 01:06:37.281 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.302 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.325 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.344 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.368 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.391 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.454 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.462 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.480 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.492 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.505 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.522 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.533 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.541 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.549 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:37.557 UTC [12350] WARNING: there is no transaction in progress
2026-09-26 01:06:38,400 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 01:06:38,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 01:06:38,667 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 01:06:38,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 01:06:38,920 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 01:06:38,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 01:06:39,187 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 01:06:39,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 01:06:39,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 01:06:39,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-l73v.c.gcp-upstream.internal';"
2026-09-26 01:06:39,951 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 01:06:39,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 01:06:40,213 [I] DB_PATCHES have already been applied
I20260926 01:06:40.219048 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-admin:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasky1EZ9q/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 01:06:40.219118 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 01:06:40.222046 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39113
Sep 26, 2026 1:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:06:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:06:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 01:06:41.236210 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.147s user 0.001s sys 0.007s
Sep 26, 2026 1:07:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:07:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:07:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:07:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:07:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 01:07:03.366572 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 01:07:03 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384823, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 01:07:03.466816 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 01:07:03.475872 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 01:07:04.507898 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 01:07:04.507994 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-kms
I20260926 01:07:04.508011 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:33693
2026-09-26 01:07:04,659 [I] DB FLAVOR :POSTGRES
2026-09-26 01:07:04,659 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 01:07:04,659 [I] Checking connection
2026-09-26 01:07:04,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 01:07:04,936 [I] connection success
2026-09-26 01:07:04,937 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 01:07:04,937 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 01:07:04,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 01:07:05,186 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 01:07:05,186 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 01:07:05,186 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 01:07:05,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33693/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 01:07:05,463 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 01:07:05.998131 392 mini_ranger_kms.cc:310] Created kms service
I20260926 01:07:06.187985 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 01:07:06.243762 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 01:07:06 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384823, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:07:06.736634 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 01:07:06.736855 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-kms:/tmp/dist-test-tasky1EZ9q/build/release/bin/postgresql.jar:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasky1EZ9q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 01:07:06.736877 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 01:07:06.740455 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12531
I20260926 01:07:06.740545 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:38309
Sep 26, 2026 1:07:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:07:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:07:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:07:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:07:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:07:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 01:07:08.812198 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.345s user 0.005s sys 0.128s
I20260926 01:07:08.812284 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 01:07:08.812326 392 mini_ranger_kms.cc:401] 127.0.98.20:38309/kms/v1/keys
Sep 26 01:07:08 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790384828, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 01:07:09 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384828, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 01:07:09 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384828, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 01:07:12 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:07:12.306761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33741
--ranger_config_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38309/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:12.389209 12581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:12.389461 12581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:12.389496 12581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:12.390899 12581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:07:12.390969 12581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:12.390986 12581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:12.391002 12581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:07:12.391017 12581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:07:12.392932 12581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:38309/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33741
--ranger_config_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:12.393234 12581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:12.393450 12581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:12.399322 12589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:12.399322 12587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:12.399590 12581 server_base.cc:1034] running on GCE node
W20260926 01:07:12.399349 12586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:12.399937 12581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:12.400259 12581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:12.401405 12581 hybrid_clock.cc:648] HybridClock initialized: now 1790384832401379 us; error 45 us; skew 500 ppm
Sep 26 01:07:12 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:12.408191 12581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:07:12.408761 12581 webserver.cc:460] Webserver started at http://127.0.98.62:34507/ using document root <none> and password file <none>
I20260926 01:07:12.408972 12581 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:12.409014 12581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:12.409157 12581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:07:12 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:07:12.466516 12581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "fdb300c3e4b74c2fbc81028e2277e253"
format_stamp: "Formatted at 2026-09-26 01:07:12 on dist-test-slave-l73v"
server_key: "e591dd89244e4def4662ec053f4dc0b3"
server_key_iv: "12c00bac737891263e998767e5dcf672"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:12.466959 12581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdb300c3e4b74c2fbc81028e2277e253"
format_stamp: "Formatted at 2026-09-26 01:07:12 on dist-test-slave-l73v"
server_key: "e591dd89244e4def4662ec053f4dc0b3"
server_key_iv: "12c00bac737891263e998767e5dcf672"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:12.468647 12581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:07:12.481220 12596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:12.481545 12581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:12.481719 12581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "fdb300c3e4b74c2fbc81028e2277e253"
format_stamp: "Formatted at 2026-09-26 01:07:12 on dist-test-slave-l73v"
server_key: "e591dd89244e4def4662ec053f4dc0b3"
server_key_iv: "12c00bac737891263e998767e5dcf672"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:12.481832 12581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:12.498543 12581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:12.500898 12581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:12.501057 12581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:12.505985 12581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33741
I20260926 01:07:12.506013 12648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33741 every 8 connection(s)
I20260926 01:07:12.506467 12581 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:07:12.507237 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:12.509591 12649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: Bootstrap starting.
I20260926 01:07:12.510190 12649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:12.510507 12649 log.cc:824] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:12.511234 12649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: No bootstrap required, opened a new log
I20260926 01:07:12.513445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 12581
I20260926 01:07:12.513566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:07:12.513767 392 external_mini_cluster.cc:1428] Setting key cfbbf7a30e6467c56c48c62f1567ea99
I20260926 01:07:12.514197 12649 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 01:07:12.514685 12649 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:12.514751 12649 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb300c3e4b74c2fbc81028e2277e253, State: Initialized, Role: FOLLOWER
I20260926 01:07:12.514917 12649 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 01:07:12.515100 12649 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:07:12.515153 12649 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:07:12.515197 12649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:12.516012 12649 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 01:07:12.516119 12649 leader_election.cc:302] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdb300c3e4b74c2fbc81028e2277e253; no voters:
I20260926 01:07:12.516320 12649 leader_election.cc:288] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:07:12.516364 12654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:07:12.516575 12654 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 1 LEADER]: Becoming Leader. State: Replica: fdb300c3e4b74c2fbc81028e2277e253, State: Running, Role: LEADER
I20260926 01:07:12.516678 12649 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:07:12.516732 12654 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 01:07:12.517242 12655 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdb300c3e4b74c2fbc81028e2277e253. Latest consensus state: current_term: 1 leader_uuid: "fdb300c3e4b74c2fbc81028e2277e253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } } }
I20260926 01:07:12.517396 12655 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:12.517674 12663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:07:12.518234 12663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:07:12.518533 12649 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 01:07:12.517102 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdb300c3e4b74c2fbc81028e2277e253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } } }
I20260926 01:07:12.519361 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [sys.catalog]: This master's current role is: LEADER
Sep 26 01:07:12 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:12.526696 12663 catalog_manager.cc:1347] Generated new cluster ID: 7610db6051ac484ea4863b036d9bfea4
I20260926 01:07:12.526822 12663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:07:12.528959 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:12.514306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.0.98.62:33741)
0926 01:07:12.514557 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:12.514568 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:12.514609 (+ 41us) server_negotiation.cc:407] Connection header received
0926 01:07:12.514742 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:12.514750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:12.514819 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:12.514966 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:12.515708 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:12.517093 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:12.518189 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:12.518825 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:12.518924 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:12.521896 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:12.521901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:12.521901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:12.521915 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:12.526371 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:12.526991 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:12.526995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:12.526995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:12.527570 (+ 575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:12.527789 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:12.527790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:12.527791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:12.528135 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:12.528332 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:12.528404 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 01:07:12.528732 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":136,"threads_started":1}
I20260926 01:07:12.532799 12663 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:07:12.533654 12663 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:07:12.540560 12663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: Generated new TSK 0
I20260926 01:07:12.540872 12663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:07:13.365836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:13.490479 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:13.490768 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:13.490808 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:13.492699 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:13.492775 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:13.492822 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:07:13.495270 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:38309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:13.495579 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:13.495900 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:13.501411 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:13.501567 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:13.501492 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:13.501775 12694 server_base.cc:1034] running on GCE node
I20260926 01:07:13.502034 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:13.502287 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:13.503437 12694 hybrid_clock.cc:648] HybridClock initialized: now 1790384833503424 us; error 26 us; skew 500 ppm
Sep 26 01:07:13 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:13.506978 12694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:07:13.507570 12694 webserver.cc:460] Webserver started at http://127.0.98.1:38801/ using document root <none> and password file <none>
I20260926 01:07:13.507855 12694 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:13.507925 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:13.508069 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:07:13 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:07:13.526311 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "863db358cefa53716c81567651925154"
server_key_iv: "da134ce4c3ccb4fabaee76ab6a5356fd"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:13.526763 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "863db358cefa53716c81567651925154"
server_key_iv: "da134ce4c3ccb4fabaee76ab6a5356fd"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:13.528101 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:13.541535 12710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:13.541729 12694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:13.541822 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "863db358cefa53716c81567651925154"
server_key_iv: "da134ce4c3ccb4fabaee76ab6a5356fd"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:13.541929 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:13.553628 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:13.556233 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:13.556386 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:13.556644 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:13.557017 12694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:13.557071 12694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:13.557153 12694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:13.557201 12694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:13.563789 12694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43325
I20260926 01:07:13.564302 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:07:13.573982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 12694
I20260926 01:07:13.574095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:07:13.574297 392 external_mini_cluster.cc:1428] Setting key ac179972e4d0795b46ab7c5c7bb87b7e
I20260926 01:07:13.575204 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43325 every 8 connection(s)
Sep 26 01:07:13 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:07:13.588286 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:13.577246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33535 (local address 127.0.98.62:33741)
0926 01:07:13.577374 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:13.577376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:13.577630 (+ 254us) server_negotiation.cc:407] Connection header received
0926 01:07:13.578350 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:13.578352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:13.578366 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:13.578443 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:13.579434 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:13.580203 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:13.581562 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:13.581830 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:13.581894 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:13.584550 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:13.584552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:13.584552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:13.584562 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:13.586446 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:13.587049 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:13.587051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:13.587051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:13.587120 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:13.587459 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:13.587461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:13.587461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:13.587613 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:13.587659 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:13.588069 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 01:07:13.588174 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
I20260926 01:07:13.588784 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:13.575905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33741 (local address 127.0.98.1:33535)
0926 01:07:13.577462 (+ 1557us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:13.577521 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:13.578225 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:13.578670 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:13.578674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:13.579008 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:13.579307 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:13.579314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:13.580330 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:13.580332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:13.581460 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:13.581463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:13.581987 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:13.581989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:13.582136 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:13.582770 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:13.582781 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:13.584416 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:13.586560 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:13.586571 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:13.586574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:13.586952 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:13.587217 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:13.587218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:13.587218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:13.587375 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:13.587710 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:13.587717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:13.587914 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 01:07:13.588387 (+ 473us) client_negotiation.cc:240] Negotiation successful
0926 01:07:13.588564 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1503,"thread_start_us":107,"threads_started":1}
I20260926 01:07:13.589331 12825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33741
I20260926 01:07:13.589430 12825 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:13.589629 12825 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 01:07:13.590111 12613 ts_manager.cc:194] Registered new tserver with Master: 63a9f7fad0ca499ba3faee45cd3fdf4a (127.0.98.1:43325)
I20260926 01:07:13.590831 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:07:13.605767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:13.709559 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:13.709795 12831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:13.709832 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:13.711633 12831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:13.711695 12831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:13.711738 12831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:07:13.713649 12831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:38309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:13.713923 12831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:13.714222 12831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:13.719429 12839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:13.719379 12838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:13.719779 12841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:13.720250 12831 server_base.cc:1034] running on GCE node
I20260926 01:07:13.720464 12831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:13.720741 12831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:13.721925 12831 hybrid_clock.cc:648] HybridClock initialized: now 1790384833721903 us; error 33 us; skew 500 ppm
Sep 26 01:07:13 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:13.724694 12831 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:07:13.725222 12831 webserver.cc:460] Webserver started at http://127.0.98.2:38101/ using document root <none> and password file <none>
I20260926 01:07:13.725515 12831 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:13.725594 12831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:13.725785 12831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:07:13 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 01:07:13 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790384833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:13.741830 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "a2cc546968fa4f25a1d26088610cea4b"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "dd6ee00878b08beaeb81abe94e241651"
server_key_iv: "e92ce0c85c435c96a1b811d6837d3555"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:13.742359 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a2cc546968fa4f25a1d26088610cea4b"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "dd6ee00878b08beaeb81abe94e241651"
server_key_iv: "e92ce0c85c435c96a1b811d6837d3555"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:13.746019 12831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 01:07:13.761695 12848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:13.762714 12831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 01:07:13.762897 12831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "a2cc546968fa4f25a1d26088610cea4b"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "dd6ee00878b08beaeb81abe94e241651"
server_key_iv: "e92ce0c85c435c96a1b811d6837d3555"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:13.763093 12831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:13.777906 12831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:13.780637 12831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:13.780858 12831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:13.781126 12831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:13.781616 12831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:13.781708 12831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:13.781790 12831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:13.781848 12831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:13.788457 12831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41583
I20260926 01:07:13.788986 12831 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:07:13.792382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 12831
I20260926 01:07:13.792498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:07:13.792594 12961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41583 every 8 connection(s)
I20260926 01:07:13.792707 392 external_mini_cluster.cc:1428] Setting key f744ca22529aa1c0c1ab81c3640e3c7b
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 26 01:07:13 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 01:07:13.812139 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:13.793650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37835 (local address 127.0.98.62:33741)
0926 01:07:13.793715 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:13.793717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:13.798739 (+ 5022us) server_negotiation.cc:407] Connection header received
0926 01:07:13.799521 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:13.799523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:13.799538 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:13.799569 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:13.800448 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:13.801246 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:13.802854 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:13.803165 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:13.803219 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:13.808470 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:13.808472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:13.808473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:13.808484 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:13.810225 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:13.810904 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:13.810906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:13.810906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:13.810973 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:13.811305 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:13.811306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:13.811306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:13.811457 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:13.811506 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:13.811932 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 01:07:13.812033 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:07:13.812770 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:13.793352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33741 (local address 127.0.98.2:37835)
0926 01:07:13.798546 (+ 5194us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:13.798617 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:13.799386 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:13.799772 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:13.799776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:13.800036 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:13.800332 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:13.800339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:13.801373 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:13.801375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:13.802749 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:13.802752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:13.803334 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:13.803335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:13.803488 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:13.804125 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:13.804136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:13.807661 (+ 3525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:13.810351 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:13.810365 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:13.810368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:13.810788 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:13.811045 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:13.811046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:13.811046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:13.811225 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:13.811538 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:13.811546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:13.811752 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 01:07:13.812260 (+ 508us) client_negotiation.cc:240] Negotiation successful
0926 01:07:13.812458 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5127,"thread_start_us":99,"threads_started":1}
I20260926 01:07:13.813490 12962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33741
I20260926 01:07:13.813601 12962 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:13.813819 12962 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 01:07:13.814254 12613 ts_manager.cc:194] Registered new tserver with Master: a2cc546968fa4f25a1d26088610cea4b (127.0.98.2:41583)
I20260926 01:07:13.814780 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:07:13.829759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:13.950556 12967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:13.950872 12967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:13.950914 12967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:13.952991 12967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:13.953121 12967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:13.953197 12967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:07:13.955777 12967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:38309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:13.956236 12967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:13.956538 12967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:13.963224 12976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:13.963224 12974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:13.964202 12973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:13.964856 12967 server_base.cc:1034] running on GCE node
I20260926 01:07:13.965140 12967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:13.965466 12967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:13.966640 12967 hybrid_clock.cc:648] HybridClock initialized: now 1790384833966613 us; error 34 us; skew 500 ppm
Sep 26 01:07:13 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:13.970100 12967 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:07:13.970702 12967 webserver.cc:460] Webserver started at http://127.0.98.3:44155/ using document root <none> and password file <none>
I20260926 01:07:13.970978 12967 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:13.971037 12967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:13.971222 12967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 01:07:13 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:07:13.986236 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "17a3f0d15fd9473089f446232b890082"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "ad7949d17f71c329eb8e88e7d27cd04c"
server_key_iv: "c4d150eb2ff934a1f3af456e5f1cf936"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:13.986864 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "17a3f0d15fd9473089f446232b890082"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "ad7949d17f71c329eb8e88e7d27cd04c"
server_key_iv: "c4d150eb2ff934a1f3af456e5f1cf936"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:13.994071 12967 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260926 01:07:14.026284 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.028972 12967 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260926 01:07:14.029217 12967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "17a3f0d15fd9473089f446232b890082"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "ad7949d17f71c329eb8e88e7d27cd04c"
server_key_iv: "c4d150eb2ff934a1f3af456e5f1cf936"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:14.029438 12967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:14.040970 12967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:14.043355 12967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:14.043497 12967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:14.043745 12967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:14.044112 12967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:14.044147 12967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.044178 12967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:14.044215 12967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.051973 12967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42375
I20260926 01:07:14.052527 12967 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:07:14.053155 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42375 every 8 connection(s)
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:14.062057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 12967
I20260926 01:07:14.062163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:07:14.062341 392 external_mini_cluster.cc:1428] Setting key 875363fb555be903c1a4a2cdf856fa66
I20260926 01:07:14.062423 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:14.053546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59259 (local address 127.0.98.62:33741)
0926 01:07:14.053589 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:14.053591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:14.053601 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:07:14.054084 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:14.054085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:14.054099 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:14.054150 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:14.054928 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.055714 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.057163 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.057363 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.057424 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:14.059743 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:14.059746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:14.059746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:14.059753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:14.061014 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:14.061413 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:14.061414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:14.061414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:14.061465 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:14.061695 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:14.061696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:14.061696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:14.061806 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:14.061861 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:14.062234 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 01:07:14.062321 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:07:14.062896 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:14.053272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33741 (local address 127.0.98.3:59259)
0926 01:07:14.053419 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:14.053461 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:14.053974 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:14.054229 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:14.054233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:14.054520 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:14.054810 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:14.054818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.055813 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.055814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:14.057062 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:14.057065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.057462 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.057463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:14.057609 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:14.058168 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:14.058179 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:14.059631 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:14.061098 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:14.061106 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:14.061109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:14.061339 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:14.061520 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:14.061521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:14.061522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:14.061629 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:14.061861 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:14.061867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:14.062072 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 01:07:14.062516 (+ 444us) client_negotiation.cc:240] Negotiation successful
0926 01:07:14.062701 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260926 01:07:14.063333 13097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33741
I20260926 01:07:14.063436 13097 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:14.063620 13097 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 01:07:14.063992 12613 ts_manager.cc:194] Registered new tserver with Master: 17a3f0d15fd9473089f446232b890082 (127.0.98.3:42375)
I20260926 01:07:14.064381 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59259
I20260926 01:07:14.074585 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:07:14.075110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 12694
I20260926 01:07:14.079903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 12831
I20260926 01:07:14.085001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 12967
I20260926 01:07:14.095366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 12581
I20260926 01:07:14.100633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33741
--webserver_interface=127.0.98.62
--webserver_port=34507
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33741
--ranger_config_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38309/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38309/kms with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:14.181751 13100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:14.181984 13100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:14.182008 13100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:14.183271 13100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:07:14.183326 13100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:14.183339 13100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:14.183350 13100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:07:14.183360 13100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:07:14.185086 13100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38309/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33741
--ranger_config_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:14.185319 13100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:14.185504 13100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:07:14.190397 13100 server_base.cc:1034] running on GCE node
W20260926 01:07:14.190282 13108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:14.190282 13105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:14.190290 13106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:14.190732 13100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:14.191053 13100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:14.192219 13100 hybrid_clock.cc:648] HybridClock initialized: now 1790384834192204 us; error 29 us; skew 500 ppm
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:14.194783 13100 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:07:14.195218 13100 webserver.cc:460] Webserver started at http://127.0.98.62:34507/ using document root <none> and password file <none>
I20260926 01:07:14.195428 13100 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:14.195482 13100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:07:14.243757 13100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:14.244638 13116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.244860 13100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:14.244971 13100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "fdb300c3e4b74c2fbc81028e2277e253"
format_stamp: "Formatted at 2026-09-26 01:07:12 on dist-test-slave-l73v"
server_key: "e591dd89244e4def4662ec053f4dc0b3"
server_key_iv: "12c00bac737891263e998767e5dcf672"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:14.245461 13100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:14.265612 13100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:14.267545 13100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:14.267664 13100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:14.271646 13100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33741
I20260926 01:07:14.271687 13168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33741 every 8 connection(s)
I20260926 01:07:14.272063 13100 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:07:14.273298 13169 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:07:14.273962 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: Bootstrap starting.
I20260926 01:07:14.276240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 13100
I20260926 01:07:14.276501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43325
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33741
--webserver_port=38801
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38309/kms with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:07:14.281792 13169 log.cc:824] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:14.283555 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:07:14.283839 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: Bootstrap complete.
I20260926 01:07:14.286043 13169 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 01:07:14.286293 13169 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb300c3e4b74c2fbc81028e2277e253, State: Initialized, Role: FOLLOWER
I20260926 01:07:14.286481 13169 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 01:07:14.286564 13169 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:07:14.286615 13169 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:07:14.286662 13169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:07:14.287776 13169 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 01:07:14.287879 13169 leader_election.cc:302] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdb300c3e4b74c2fbc81028e2277e253; no voters:
I20260926 01:07:14.288040 13169 leader_election.cc:288] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:07:14.288131 13173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:07:14.288280 13173 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [term 2 LEADER]: Becoming Leader. State: Replica: fdb300c3e4b74c2fbc81028e2277e253, State: Running, Role: LEADER
I20260926 01:07:14.288374 13169 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:07:14.288439 13173 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 01:07:14.288730 13174 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fdb300c3e4b74c2fbc81028e2277e253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } } }
I20260926 01:07:14.288805 13174 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:14.289068 13175 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdb300c3e4b74c2fbc81028e2277e253. Latest consensus state: current_term: 2 leader_uuid: "fdb300c3e4b74c2fbc81028e2277e253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb300c3e4b74c2fbc81028e2277e253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } } }
I20260926 01:07:14.289117 13175 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253 [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:14.289383 13179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:07:14.289812 13169 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 01:07:14.290023 13179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:07:14.290258 13179 catalog_manager.cc:1259] Loaded cluster ID: 7610db6051ac484ea4863b036d9bfea4
I20260926 01:07:14.290329 13179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:07:14.294343 13179 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:07:14.294905 13179 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: Loaded TSK: 0
I20260926 01:07:14.295212 13179 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:07:14.384594 13171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:14.384825 13171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:14.384859 13171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:14.386145 13171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:14.386197 13171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:14.386234 13171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:07:14.388077 13171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38801
--tserver_master_addrs=127.0.98.62:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:14.388356 13171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:14.388622 13171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:14.395927 13210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:14.395927 13207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:14.397665 13171 server_base.cc:1034] running on GCE node
W20260926 01:07:14.398448 13208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:14.398765 13171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:14.399080 13171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:14.400246 13171 hybrid_clock.cc:648] HybridClock initialized: now 1790384834400225 us; error 36 us; skew 500 ppm
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:14.403002 13171 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:07:14.403496 13171 webserver.cc:460] Webserver started at http://127.0.98.1:38801/ using document root <none> and password file <none>
I20260926 01:07:14.403739 13171 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:14.403803 13171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:07:14.433840 13171 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.001s
I20260926 01:07:14.434846 13218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.435060 13171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:07:14.435145 13171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "863db358cefa53716c81567651925154"
server_key_iv: "da134ce4c3ccb4fabaee76ab6a5356fd"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:14.435612 13171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:14.459152 13171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:14.461162 13171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:14.461310 13171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:14.461511 13171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:14.461851 13171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:14.461895 13171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.461941 13171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:14.461966 13171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.468479 13171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43325
I20260926 01:07:14.468572 13331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43325 every 8 connection(s)
I20260926 01:07:14.468895 13171 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:07:14.472153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 13171
I20260926 01:07:14.472433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41583
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33741
--webserver_port=38101
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38309/kms with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:14.485828 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:14.469623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33741 (local address 127.0.98.1:45491)
0926 01:07:14.469774 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:14.469826 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:14.470499 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:14.470983 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:14.470988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:14.471249 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:14.471533 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:14.471540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.473018 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.473019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:14.474326 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:14.474330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.475202 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.475203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:14.475357 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:14.475985 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:14.475996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:14.481209 (+ 5213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:14.483901 (+ 2692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:14.483913 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:14.483915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:14.484244 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:14.484552 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:14.484553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:14.484553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:14.484679 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:14.485139 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:14.485146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:14.485313 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 01:07:14.485406 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 01:07:14.485588 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260926 01:07:14.486810 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:14.469770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45491 (local address 127.0.98.62:33741)
0926 01:07:14.469964 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:14.469973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:14.470005 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:07:14.470735 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:14.470742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:14.470789 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:14.470898 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:14.471650 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.472900 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.474459 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.475081 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.475193 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:14.481412 (+ 6219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:14.481430 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:14.481430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:14.481449 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:14.483775 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:14.484342 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:14.484346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:14.484347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:14.484484 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:14.484740 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:14.484741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:14.484741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:14.485041 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:14.485112 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:14.486270 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0926 01:07:14.486430 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260926 01:07:14.487684 13332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33741
I20260926 01:07:14.487780 13332 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:14.487974 13332 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
W20260926 01:07:14.488979 13332 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33741 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 01:07:14.489046 13332 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:14.489109 13332 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 01:07:14.489665 13332 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:14.489735 13332 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
W20260926 01:07:14.490069 13332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:07:14.578058 13336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:14.578349 13336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:14.578413 13336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:14.579797 13336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:14.579849 13336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:14.579891 13336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:07:14.581549 13336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38101
--tserver_master_addrs=127.0.98.62:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:14.581808 13336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:14.582077 13336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:14.588518 13344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:14.590166 13342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:14.591373 13341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:14.592344 13336 server_base.cc:1034] running on GCE node
I20260926 01:07:14.592583 13336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:14.592864 13336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:14.594039 13336 hybrid_clock.cc:648] HybridClock initialized: now 1790384834594021 us; error 26 us; skew 500 ppm
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:14.596829 13336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:07:14.597420 13336 webserver.cc:460] Webserver started at http://127.0.98.2:38101/ using document root <none> and password file <none>
I20260926 01:07:14.597692 13336 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:14.597791 13336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:07:14.617246 13336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:14.618000 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.618144 13336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.618220 13336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "a2cc546968fa4f25a1d26088610cea4b"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "dd6ee00878b08beaeb81abe94e241651"
server_key_iv: "e92ce0c85c435c96a1b811d6837d3555"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:14.618744 13336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:14.654870 13336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:14.657189 13336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:14.657344 13336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:14.657613 13336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:14.657989 13336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:14.658036 13336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.658066 13336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:14.658088 13336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.665880 13336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41583
I20260926 01:07:14.665963 13465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41583 every 8 connection(s)
I20260926 01:07:14.666404 13336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:07:14.668725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 13336
I20260926 01:07:14.669044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42375
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33741
--webserver_port=44155
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38309/kms with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:14.678689 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:14.667136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51725 (local address 127.0.98.62:33741)
0926 01:07:14.667181 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:14.667183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:14.667444 (+ 261us) server_negotiation.cc:407] Connection header received
0926 01:07:14.668112 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:14.668113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:14.668130 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:14.668162 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:14.669019 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.669772 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.671346 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.671633 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.671703 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:14.674449 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:14.674452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:14.674452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:14.674462 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:14.676794 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:14.677398 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:14.677400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:14.677401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:14.677468 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:14.677784 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:14.677785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:14.677785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:14.677938 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:14.677986 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:14.678480 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 01:07:14.678585 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:07:14.679371 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:14.667131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33741 (local address 127.0.98.2:51725)
0926 01:07:14.667290 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:14.667338 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:14.667977 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:14.668350 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:14.668356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:14.668589 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:14.668905 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:14.668912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.669906 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.669906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:14.671236 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:14.671240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.671751 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.671752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:14.671921 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:14.672598 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:14.672611 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:14.674309 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:14.676912 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:14.676924 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:14.676927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:14.677281 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:14.677545 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:14.677546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:14.677546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:14.677702 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:14.678038 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:14.678046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:14.678262 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 01:07:14.678842 (+ 580us) client_negotiation.cc:240] Negotiation successful
0926 01:07:14.679041 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260926 01:07:14.679819 13466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33741
I20260926 01:07:14.679922 13466 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:14.680116 13466 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
W20260926 01:07:14.680714 13466 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33741 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 01:07:14.680783 13466 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:14.680837 13466 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 01:07:14.681095 13466 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:14.681160 13466 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
W20260926 01:07:14.681394 13466 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:07:14.772691 13469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:14.772930 13469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:14.772965 13469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:14.774288 13469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:14.774348 13469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:14.774415 13469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:07:14.776131 13469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44155
--tserver_master_addrs=127.0.98.62:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:14.776407 13469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:14.776634 13469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:14.781710 13477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:14.781601 13475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:14.781601 13474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:14.782243 13469 server_base.cc:1034] running on GCE node
I20260926 01:07:14.782544 13469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:14.782805 13469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:14.783962 13469 hybrid_clock.cc:648] HybridClock initialized: now 1790384834783945 us; error 27 us; skew 500 ppm
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:14.786540 13469 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:07:14.787058 13469 webserver.cc:460] Webserver started at http://127.0.98.3:44155/ using document root <none> and password file <none>
I20260926 01:07:14.787292 13469 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:14.787355 13469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:07:14.815667 13469 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260926 01:07:14.816473 13486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.816615 13469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:07:14.816700 13469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "17a3f0d15fd9473089f446232b890082"
format_stamp: "Formatted at 2026-09-26 01:07:13 on dist-test-slave-l73v"
server_key: "ad7949d17f71c329eb8e88e7d27cd04c"
server_key_iv: "c4d150eb2ff934a1f3af456e5f1cf936"
server_key_version: "kuduclusterkey@0"
I20260926 01:07:14.817104 13469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:14.837203 13469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:14.839705 13469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:14.839877 13469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:14.840127 13469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:14.840499 13469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:14.840547 13469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.840582 13469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:14.840598 13469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:14.847662 13469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42375
I20260926 01:07:14.848255 13469 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:07:14.848800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 13469
I20260926 01:07:14.850423 13599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42375 every 8 connection(s)
Sep 26 01:07:14 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:14.865517 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:14.849302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.0.98.62:33741)
0926 01:07:14.849358 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:14.849360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:14.854680 (+ 5320us) server_negotiation.cc:407] Connection header received
0926 01:07:14.854762 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:14.854764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:14.854780 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:14.854843 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:14.856250 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.857011 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.858147 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.858474 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.858546 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:14.861304 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:14.861306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:14.861307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:14.861317 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:14.863082 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:14.864474 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:14.864475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:14.864476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:14.864548 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:14.864743 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:14.864743 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:14.864744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:14.864888 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:14.865054 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:14.865321 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 01:07:14.865417 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:07:14.869362 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:14.850695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33741 (local address 127.0.98.3:47353)
0926 01:07:14.850897 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:14.850947 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:14.851591 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:14.855892 (+ 4301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:14.855899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:14.856152 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:14.856462 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:14.856470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.860119 (+ 3649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.860120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:14.861550 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:14.861555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.862750 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.862751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:14.862923 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:14.863483 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:14.863494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:14.865064 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:14.867658 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:14.867669 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:14.867671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:14.867999 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:14.868210 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:14.868211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:14.868211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:14.868354 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:14.868621 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:14.868627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:14.868817 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 01:07:14.868918 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 01:07:14.869088 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260926 01:07:14.869746 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:14.850653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47353 (local address 127.0.98.62:33741)
0926 01:07:14.854779 (+ 4126us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:14.854781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:14.854790 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:07:14.854812 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:14.854813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:14.855661 (+ 848us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:14.856567 (+ 906us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:14.856606 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.858467 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.861993 (+ 3526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:14.862621 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:14.864494 (+ 1873us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:14.865608 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:14.865609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:14.865610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:14.865618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:14.867545 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:14.868082 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:14.868083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:14.868083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:14.868143 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:14.868417 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:14.868417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:14.868418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:14.868554 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:14.869505 (+ 951us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:14.869555 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 01:07:14.869648 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4101,"thread_start_us":59,"threads_started":1}
I20260926 01:07:14.870893 13600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33741
I20260926 01:07:14.871013 13600 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:14.871227 13600 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
W20260926 01:07:14.871912 13600 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33741 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 01:07:14.872011 13600 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:14.872084 13600 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 01:07:14.872388 13600 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:14.872464 13600 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
W20260926 01:07:14.872707 13600 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 01:07:15 dist-test-slave-l73v krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790384835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:15.490237 13332 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:15.490432 13332 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 01:07:15.490741 13133 ts_manager.cc:194] Registered new tserver with Master: 63a9f7fad0ca499ba3faee45cd3fdf4a (127.0.98.1:43325)
I20260926 01:07:15.491590 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45491
I20260926 01:07:15.681591 13466 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:15.681758 13466 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 01:07:15.682089 13133 ts_manager.cc:194] Registered new tserver with Master: a2cc546968fa4f25a1d26088610cea4b (127.0.98.2:41583)
I20260926 01:07:15.682588 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51725
Sep 26 01:07:15 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:07:15.872884 13600 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:15.873162 13600 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 01:07:15.873502 13133 ts_manager.cc:194] Registered new tserver with Master: 17a3f0d15fd9473089f446232b890082 (127.0.98.3:42375)
I20260926 01:07:15.874034 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47353
I20260926 01:07:15.882529 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:07:15.892693 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:15.883604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.0.98.62:33741)
0926 01:07:15.883894 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:15.883896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:15.884059 (+ 163us) server_negotiation.cc:407] Connection header received
0926 01:07:15.884085 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:15.884086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:15.884098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:15.884162 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:15.884886 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.885701 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.886809 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.887137 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.887200 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:15.889104 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:15.889107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:15.889107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:15.889116 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:15.890995 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:15.891410 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:15.891411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:15.891412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:15.891489 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:15.891703 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:15.891704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:15.891704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:15.891852 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:15.891948 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:15.892480 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 01:07:15.892577 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":68,"threads_started":1}
I20260926 01:07:15.895777 13133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:07:15.898241 13133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:07:15.913344 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:15.907107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.0.98.2:41583)
0926 01:07:15.907235 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:15.907240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:15.908088 (+ 848us) server_negotiation.cc:407] Connection header received
0926 01:07:15.908194 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:15.908197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:15.908214 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:15.908264 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:15.908556 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.909683 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.911611 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.912461 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.912515 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:15.912600 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:15.913189 (+ 589us) server_negotiation.cc:299] Negotiation successful
0926 01:07:15.913241 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":45,"threads_started":1}
I20260926 01:07:15.913892 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:15.907016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41583 (local address 127.0.0.1:38852)
0926 01:07:15.908021 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:15.908031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:15.908117 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:15.908379 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:15.908382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:15.908384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:15.908476 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:15.908477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.910063 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.910064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:15.911493 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:15.911497 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.912954 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.912955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:15.913062 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:15.913077 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:07:15.913721 (+ 644us) client_negotiation.cc:240] Negotiation successful
0926 01:07:15.913807 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":84,"threads_started":1}
I20260926 01:07:15.914944 13392 tablet_service.cc:1467] Processing CreateTablet for tablet fb6e3f95d41444a5abca93c773808082 (DEFAULT_TABLE table=test-table [id=dffabca00c954aa197d7dd3db23c820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:15.915344 13392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6e3f95d41444a5abca93c773808082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:15.919482 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:15.907408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.0.98.1:43325)
0926 01:07:15.907929 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:15.907934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:15.914206 (+ 6272us) server_negotiation.cc:407] Connection header received
0926 01:07:15.914313 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:15.914317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:15.914332 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:15.914404 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:15.914607 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.916175 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.917525 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.918558 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.918626 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:15.918704 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:15.919337 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 01:07:15.919388 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":61,"threads_started":1}
I20260926 01:07:15.919723 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:15.907220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43325 (local address 127.0.0.1:52462)
0926 01:07:15.914137 (+ 6917us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:15.914148 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:15.914235 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:15.914504 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:15.914505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:15.914506 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:15.914537 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:15.914538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.916263 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.916264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:15.917436 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:15.917439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.918769 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.918770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:15.918837 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:15.918840 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:07:15.919587 (+ 747us) client_negotiation.cc:240] Negotiation successful
0926 01:07:15.919650 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6906,"thread_start_us":63,"threads_started":1}
I20260926 01:07:15.919888 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:15.906992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42375 (local address 127.0.0.1:36594)
0926 01:07:15.907300 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:15.907339 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:15.907406 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:15.913767 (+ 6361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:15.913768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:15.913771 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:15.913822 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:15.913823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.915255 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.915257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:15.916375 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:15.916377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.919678 (+ 3301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.919680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:15.919749 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:15.919753 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:07:15.919783 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 01:07:15.919821 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":81,"threads_started":1}
I20260926 01:07:15.920478 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:15.907088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.0.98.3:42375)
0926 01:07:15.913576 (+ 6488us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:15.913583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:15.913594 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:07:15.913630 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:15.913633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:15.913650 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:15.913896 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:15.913981 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.915132 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.916469 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:15.917240 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:15.920203 (+ 2963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:15.920317 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:15.920346 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 01:07:15.920400 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6439,"thread_start_us":45,"threads_started":1}
I20260926 01:07:15.920629 13266 tablet_service.cc:1467] Processing CreateTablet for tablet fb6e3f95d41444a5abca93c773808082 (DEFAULT_TABLE table=test-table [id=dffabca00c954aa197d7dd3db23c820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:15.921025 13266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6e3f95d41444a5abca93c773808082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:15.921116 13534 tablet_service.cc:1467] Processing CreateTablet for tablet fb6e3f95d41444a5abca93c773808082 (DEFAULT_TABLE table=test-table [id=dffabca00c954aa197d7dd3db23c820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:15.921402 13534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6e3f95d41444a5abca93c773808082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:15.925681 13635 tablet_bootstrap.cc:492] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: Bootstrap starting.
I20260926 01:07:15.926519 13635 tablet_bootstrap.cc:654] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:15.926978 13635 log.cc:824] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:15.927450 13636 tablet_bootstrap.cc:492] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a: Bootstrap starting.
I20260926 01:07:15.928256 13636 tablet_bootstrap.cc:654] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:15.928650 13636 log.cc:824] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:15.929555 13635 tablet_bootstrap.cc:492] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: No bootstrap required, opened a new log
I20260926 01:07:15.929682 13635 ts_tablet_manager.cc:1397] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 01:07:15.933001 13635 raft_consensus.cc:348] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "a2cc546968fa4f25a1d26088610cea4b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 } }
I20260926 01:07:15.933368 13635 raft_consensus.cc:374] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:15.933450 13635 raft_consensus.cc:729] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a3f0d15fd9473089f446232b890082, State: Initialized, Role: FOLLOWER
I20260926 01:07:15.933674 13635 consensus_queue.cc:260] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "a2cc546968fa4f25a1d26088610cea4b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 } }
I20260926 01:07:15.934080 13635 ts_tablet_manager.cc:1428] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260926 01:07:15.934237 13600 heartbeater.cc:503] Master 127.0.98.62:33741 was elected leader, sending a full tablet report...
I20260926 01:07:15.934581 13634 tablet_bootstrap.cc:492] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: Bootstrap starting.
I20260926 01:07:15.935333 13634 tablet_bootstrap.cc:654] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:15.935657 13634 log.cc:824] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:15.935984 13636 tablet_bootstrap.cc:492] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a: No bootstrap required, opened a new log
I20260926 01:07:15.936055 13636 ts_tablet_manager.cc:1397] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260926 01:07:15.939188 13636 raft_consensus.cc:348] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "a2cc546968fa4f25a1d26088610cea4b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 } }
I20260926 01:07:15.939486 13636 raft_consensus.cc:374] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:15.939606 13636 raft_consensus.cc:729] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a9f7fad0ca499ba3faee45cd3fdf4a, State: Initialized, Role: FOLLOWER
I20260926 01:07:15.939815 13636 consensus_queue.cc:260] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "a2cc546968fa4f25a1d26088610cea4b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 } }
I20260926 01:07:15.940313 13636 ts_tablet_manager.cc:1428] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 01:07:15.940506 13332 heartbeater.cc:503] Master 127.0.98.62:33741 was elected leader, sending a full tablet report...
I20260926 01:07:15.943204 13634 tablet_bootstrap.cc:492] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: No bootstrap required, opened a new log
I20260926 01:07:15.943295 13634 ts_tablet_manager.cc:1397] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260926 01:07:15.945986 13634 raft_consensus.cc:348] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "a2cc546968fa4f25a1d26088610cea4b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 } }
I20260926 01:07:15.946230 13634 raft_consensus.cc:374] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:15.946262 13634 raft_consensus.cc:729] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cc546968fa4f25a1d26088610cea4b, State: Initialized, Role: FOLLOWER
I20260926 01:07:15.946467 13634 consensus_queue.cc:260] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "a2cc546968fa4f25a1d26088610cea4b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 } }
I20260926 01:07:15.946799 13634 ts_tablet_manager.cc:1428] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:07:15.946924 13466 heartbeater.cc:503] Master 127.0.98.62:33741 was elected leader, sending a full tablet report...
W20260926 01:07:15.970463 13333 tablet.cc:2369] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:07:16.063030 13642 raft_consensus.cc:482] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:07:16.063205 13642 raft_consensus.cc:504] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "a2cc546968fa4f25a1d26088610cea4b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 } }
I20260926 01:07:16.063621 13642 leader_election.cc:288] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17a3f0d15fd9473089f446232b890082 (127.0.98.3:42375), 63a9f7fad0ca499ba3faee45cd3fdf4a (127.0.98.1:43325)
I20260926 01:07:16.067730 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:16.064081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39557 (local address 127.0.98.3:42375)
0926 01:07:16.064144 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:16.064146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:16.064156 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:07:16.064186 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:16.064186 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:16.064197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:16.064259 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:16.064497 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:16.065260 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:16.066765 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:16.067341 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:16.067390 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:16.067400 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:16.067608 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 01:07:16.067653 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:07:16.068017 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:16.063807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42375 (local address 127.0.98.2:39557)
0926 01:07:16.064048 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:16.064060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:16.064115 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:16.064315 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:16.064317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:16.064319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:16.064417 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:16.064419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:16.065368 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:16.065368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:16.066604 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:16.066607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:16.067460 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:16.067461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:16.067533 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:16.067536 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:07:16.067856 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 01:07:16.067902 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
I20260926 01:07:16.068483 13554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e3f95d41444a5abca93c773808082" candidate_uuid: "a2cc546968fa4f25a1d26088610cea4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a3f0d15fd9473089f446232b890082" is_pre_election: true
I20260926 01:07:16.068661 13554 raft_consensus.cc:2454] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cc546968fa4f25a1d26088610cea4b in term 0.
I20260926 01:07:16.068900 13355 leader_election.cc:302] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a3f0d15fd9473089f446232b890082, a2cc546968fa4f25a1d26088610cea4b; no voters:
I20260926 01:07:16.069033 13642 raft_consensus.cc:2781] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:07:16.069072 13642 raft_consensus.cc:482] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:07:16.069092 13642 raft_consensus.cc:3037] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:16.070079 13642 raft_consensus.cc:504] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "a2cc546968fa4f25a1d26088610cea4b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 } }
I20260926 01:07:16.070236 13642 leader_election.cc:288] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [CANDIDATE]: Term 1 election: Requested vote from peers 17a3f0d15fd9473089f446232b890082 (127.0.98.3:42375), 63a9f7fad0ca499ba3faee45cd3fdf4a (127.0.98.1:43325)
I20260926 01:07:16.070536 13554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e3f95d41444a5abca93c773808082" candidate_uuid: "a2cc546968fa4f25a1d26088610cea4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a3f0d15fd9473089f446232b890082"
I20260926 01:07:16.070621 13554 raft_consensus.cc:3037] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:16.071062 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:16.064214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60789 (local address 127.0.98.1:43325)
0926 01:07:16.064257 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:16.064258 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:16.064337 (+ 79us) server_negotiation.cc:407] Connection header received
0926 01:07:16.064420 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:16.064421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:16.064431 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:16.064457 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:16.064638 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:16.065346 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:16.066724 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:16.067280 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:16.067323 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:16.067335 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:16.070912 (+ 3577us) server_negotiation.cc:299] Negotiation successful
0926 01:07:16.070988 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:07:16.071298 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:16.063842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43325 (local address 127.0.98.2:60789)
0926 01:07:16.064294 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:16.064301 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:16.064356 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:16.064512 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:16.064513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:16.064514 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:16.064580 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:16.064581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:16.065438 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:16.065439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:16.066627 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:16.066629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:16.070738 (+ 4109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:16.070739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:16.070822 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:16.070827 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:07:16.071175 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 01:07:16.071222 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":8,"thread_start_us":92,"threads_started":1}
I20260926 01:07:16.071653 13286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e3f95d41444a5abca93c773808082" candidate_uuid: "a2cc546968fa4f25a1d26088610cea4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" is_pre_election: true
I20260926 01:07:16.071797 13286 raft_consensus.cc:2454] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cc546968fa4f25a1d26088610cea4b in term 0.
I20260926 01:07:16.071908 13285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e3f95d41444a5abca93c773808082" candidate_uuid: "a2cc546968fa4f25a1d26088610cea4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a"
I20260926 01:07:16.071969 13285 raft_consensus.cc:3037] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:16.072099 13554 raft_consensus.cc:2454] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cc546968fa4f25a1d26088610cea4b in term 1.
I20260926 01:07:16.072326 13355 leader_election.cc:302] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a3f0d15fd9473089f446232b890082, a2cc546968fa4f25a1d26088610cea4b; no voters:
I20260926 01:07:16.072410 13642 raft_consensus.cc:2781] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:07:16.072562 13642 raft_consensus.cc:686] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 1 LEADER]: Becoming Leader. State: Replica: a2cc546968fa4f25a1d26088610cea4b, State: Running, Role: LEADER
I20260926 01:07:16.072714 13642 consensus_queue.cc:237] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "a2cc546968fa4f25a1d26088610cea4b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 } }
I20260926 01:07:16.072773 13285 raft_consensus.cc:2454] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cc546968fa4f25a1d26088610cea4b in term 1.
I20260926 01:07:16.073585 13133 catalog_manager.cc:5495] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b reported cstate change: term changed from 0 to 1, leader changed from <none> to a2cc546968fa4f25a1d26088610cea4b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a2cc546968fa4f25a1d26088610cea4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2cc546968fa4f25a1d26088610cea4b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 } health_report { overall_health: UNKNOWN } } }
I20260926 01:07:16.093708 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:16.090162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.0.98.2:41583)
0926 01:07:16.090204 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:16.090205 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:16.090212 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:07:16.090251 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:16.090252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:16.090271 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:16.090328 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:07:16.090530 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:16.091276 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:16.092563 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:16.092858 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:16.092927 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:16.093152 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:07:16.093290 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:07:16.093360 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:16.093596 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 01:07:16.093622 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:07:16.095911 13285 raft_consensus.cc:1261] T fb6e3f95d41444a5abca93c773808082 P 63a9f7fad0ca499ba3faee45cd3fdf4a [term 1 FOLLOWER]: Refusing update from remote peer a2cc546968fa4f25a1d26088610cea4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:07:16.096333 13642 consensus_queue.cc:1035] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a9f7fad0ca499ba3faee45cd3fdf4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:07:16.096946 13554 raft_consensus.cc:1261] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082 [term 1 FOLLOWER]: Refusing update from remote peer a2cc546968fa4f25a1d26088610cea4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:07:16.097402 13645 consensus_queue.cc:1035] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a3f0d15fd9473089f446232b890082" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:07:16.099285 13648 mvcc.cc:205] Tried to move back new op lower bound from 7333416288645926912 to 7333416288554135552. Current Snapshot: MvccSnapshot[applied={T|T < 7333416288645926912}]
I20260926 01:07:16.100090 13650 mvcc.cc:205] Tried to move back new op lower bound from 7333416288645926912 to 7333416288554135552. Current Snapshot: MvccSnapshot[applied={T|T < 7333416288645926912}]
W20260926 01:07:16.100912 13601 tablet.cc:2369] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:07:16.103376 13133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:07:16.103492 13133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:07:16.104240 13133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdb300c3e4b74c2fbc81028e2277e253: Sending DeleteTablet for 3 replicas of tablet fb6e3f95d41444a5abca93c773808082
I20260926 01:07:16.104669 13392 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6e3f95d41444a5abca93c773808082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38852
I20260926 01:07:16.104902 13266 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6e3f95d41444a5abca93c773808082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52462
I20260926 01:07:16.105361 13534 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6e3f95d41444a5abca93c773808082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36594
I20260926 01:07:16.107098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 13171
I20260926 01:07:16.111824 13657 tablet_replica.cc:330] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: stopping tablet replica
I20260926 01:07:16.111989 13657 raft_consensus.cc:2229] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:07:16.112110 13657 raft_consensus.cc:2258] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:16.112388 13659 tablet_replica.cc:330] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: stopping tablet replica
I20260926 01:07:16.112527 13659 raft_consensus.cc:2229] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:07:16.112617 13659 raft_consensus.cc:2258] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:16.113211 13657 ts_tablet_manager.cc:1905] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:07:16.114964 13657 ts_tablet_manager.cc:1918] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:07:16.115039 13657 log.cc:1196] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/fb6e3f95d41444a5abca93c773808082
I20260926 01:07:16.115262 13657 ts_tablet_manager.cc:1939] T fb6e3f95d41444a5abca93c773808082 P a2cc546968fa4f25a1d26088610cea4b: Deleting consensus metadata
W20260926 01:07:16.116452 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43325 (error 108)
I20260926 01:07:16.117004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 13336
I20260926 01:07:16.118805 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:16.118673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43325 (local address 127.0.0.1:52472)
0926 01:07:16.118727 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:16.118767 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 01:07:16.118928 13118 catalog_manager.cc:4584] TS 63a9f7fad0ca499ba3faee45cd3fdf4a (127.0.98.1:43325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb6e3f95d41444a5abca93c773808082: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43325: connect: Connection refused (error 111)
I20260926 01:07:16.119681 13659 ts_tablet_manager.cc:1905] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:07:16.121357 13659 ts_tablet_manager.cc:1918] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:07:16.121440 13659 log.cc:1196] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/fb6e3f95d41444a5abca93c773808082
I20260926 01:07:16.121696 13659 ts_tablet_manager.cc:1939] T fb6e3f95d41444a5abca93c773808082 P 17a3f0d15fd9473089f446232b890082: Deleting consensus metadata
I20260926 01:07:16.122428 13119 catalog_manager.cc:4841] TS 17a3f0d15fd9473089f446232b890082 (127.0.98.3:42375): tablet fb6e3f95d41444a5abca93c773808082 (table test-table [id=dffabca00c954aa197d7dd3db23c820c]) successfully deleted
Sep 26 01:07:16 dist-test-slave-l73v krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 01:07:16.122885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 13469
I20260926 01:07:16.122987 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:16.122875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41583 (local address 127.0.0.1:38866)
0926 01:07:16.122928 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:16.122952 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:07:16.128854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 13100
I20260926 01:07:16.136866 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 01:07:16.471472 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 01:07:16.471570 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 01:07:16.806586 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 01:07:16.806 UTC [12167] LOG: received smart shutdown request
2026-09-26 01:07:16.810 UTC [12167] LOG: background worker "logical replication launcher" (PID 12176) exited with exit code 1
2026-09-26 01:07:16.810 UTC [12171] LOG: shutting down
2026-09-26 01:07:17.006 UTC [12167] LOG: database system is shut down
2026-09-26T01:07:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49828 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:07:17 dist-test-slave-l73v krb5kdc[13671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:07:17 dist-test-slave-l73v krb5kdc[13671](info): set up 2 sockets
Sep 26 01:07:17 dist-test-slave-l73v krb5kdc[13671](info): commencing operation
krb5kdc: starting...
W20260926 01:07:18.062079 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:07:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:07:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:07:18.185048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36183
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36183,127.0.98.61:44629,127.0.98.60:45525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:18.273237 13684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:18.273499 13684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:18.273531 13684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:18.274881 13684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:07:18.274940 13684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:18.274959 13684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:18.274974 13684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:07:18.274992 13684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:07:18.276765 13684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36183,127.0.98.61:44629,127.0.98.60:45525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:18.277060 13684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:18.277284 13684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:18.282582 13690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:18.282591 13692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:18.282591 13689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:18.282878 13684 server_base.cc:1034] running on GCE node
I20260926 01:07:18.283138 13684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:18.283452 13684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:18.284634 13684 hybrid_clock.cc:648] HybridClock initialized: now 1790384838284618 us; error 33 us; skew 500 ppm
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:18.291051 13684 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:07:18.291613 13684 webserver.cc:460] Webserver started at http://127.0.98.62:39519/ using document root <none> and password file <none>
I20260926 01:07:18.291836 13684 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:18.291888 13684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:18.292022 13684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:18.293152 13684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "79b7a62688d743ff85c0396323b2d15d"
format_stamp: "Formatted at 2026-09-26 01:07:18 on dist-test-slave-l73v"
server_key: "4b35b5bc64c1daa2f22584baeccf66d3"
server_key_iv: "e4a145a889a1fb6e3881c07d79a486eb"
server_key_version: "encryptionkey@0"
I20260926 01:07:18.293538 13684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "79b7a62688d743ff85c0396323b2d15d"
format_stamp: "Formatted at 2026-09-26 01:07:18 on dist-test-slave-l73v"
server_key: "4b35b5bc64c1daa2f22584baeccf66d3"
server_key_iv: "e4a145a889a1fb6e3881c07d79a486eb"
server_key_version: "encryptionkey@0"
I20260926 01:07:18.295195 13684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:07:18.296128 13699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:18.296362 13684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:18.296460 13684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "79b7a62688d743ff85c0396323b2d15d"
format_stamp: "Formatted at 2026-09-26 01:07:18 on dist-test-slave-l73v"
server_key: "4b35b5bc64c1daa2f22584baeccf66d3"
server_key_iv: "e4a145a889a1fb6e3881c07d79a486eb"
server_key_version: "encryptionkey@0"
I20260926 01:07:18.296531 13684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:18.308861 13684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:18.311136 13684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:18.311280 13684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:18.315369 13684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36183
I20260926 01:07:18.315404 13751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36183 every 8 connection(s)
I20260926 01:07:18.315745 13684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:07:18.316597 13752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:18.318225 13752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:07:18.324105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 13684
I20260926 01:07:18.324232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:07:18.324512 392 external_mini_cluster.cc:1428] Setting key 611f9f964eebf088d80fae90c6e54cf9
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:18.328771 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.319050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.0.98.62:36183)
0926 01:07:18.319248 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.319252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.319308 (+ 56us) server_negotiation.cc:407] Connection header received
0926 01:07:18.319894 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.319896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.319915 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.320015 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.320576 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.321434 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.322500 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.322846 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.322939 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.325178 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.325189 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.325189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.325198 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.327009 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.327510 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.327513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.327513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.327604 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.327887 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.327888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.327888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.328133 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.328221 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.328472 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.328589 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260926 01:07:18.328799 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.319010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36183 (local address 127.0.0.1:60070)
0926 01:07:18.319189 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.319233 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:18.319715 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:18.320030 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:18.320032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:18.320292 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:18.320468 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.320472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.321539 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.321541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.322420 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.322423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.322949 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.322957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.323028 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.323557 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:18.323566 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:18.325082 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:18.327108 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.327116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.327119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.327410 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.327677 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.327677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.327678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.327795 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.328212 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:18.328219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:18.328309 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 01:07:18.328415 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 01:07:18.328684 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":86,"thread_start_us":87,"threads_started":1}
I20260926 01:07:18.329653 13752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:07:18.330046 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.329942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44629 (local address 127.0.0.1:55114)
0926 01:07:18.329979 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.330023 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 01:07:18.330176 13700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44629: connect: Connection refused (error 111)
I20260926 01:07:18.330660 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.330578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44629 (local address 127.0.0.1:55126)
0926 01:07:18.330618 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.330643 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:07:18.330760 13752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44629: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44629: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 01:07:18.350498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44629
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36183,127.0.98.61:44629,127.0.98.60:45525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:07:18.380021 13752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } attempt: 1
I20260926 01:07:18.380561 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.380427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44629 (local address 127.0.0.1:55140)
0926 01:07:18.380491 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.380528 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 01:07:18.381063 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.380898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44629 (local address 127.0.0.1:55144)
0926 01:07:18.381006 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.381038 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260926 01:07:18.381168 13752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44629: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44629: connect: Connection refused (error 111)
W20260926 01:07:18.432955 13759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:18.433218 13759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:18.433244 13759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:18.434576 13759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:07:18.434629 13759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:18.434643 13759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:18.434655 13759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:07:18.434666 13759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:07:18.436388 13759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36183,127.0.98.61:44629,127.0.98.60:45525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:18.436628 13759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:18.436826 13759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:18.441815 13767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:18.441864 13759 server_base.cc:1034] running on GCE node
W20260926 01:07:18.441838 13765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:18.441815 13764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:18.442303 13759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:18.442639 13759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:18.443809 13759 hybrid_clock.cc:648] HybridClock initialized: now 1790384838443790 us; error 37 us; skew 500 ppm
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:18.446460 13759 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 01:07:18.446867 13759 webserver.cc:460] Webserver started at http://127.0.98.61:38535/ using document root <none> and password file <none>
I20260926 01:07:18.447068 13759 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:18.447108 13759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:18.447203 13759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:18.448238 13759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/data/instance:
uuid: "f6a66954e64b405cabdd12563463fd5f"
format_stamp: "Formatted at 2026-09-26 01:07:18 on dist-test-slave-l73v"
server_key: "e0807b149b6a79892f847f967aaba3c4"
server_key_iv: "6b2d10f0faf39533e93b3e83939f30c5"
server_key_version: "encryptionkey@0"
I20260926 01:07:18.448529 13759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/wal/instance:
uuid: "f6a66954e64b405cabdd12563463fd5f"
format_stamp: "Formatted at 2026-09-26 01:07:18 on dist-test-slave-l73v"
server_key: "e0807b149b6a79892f847f967aaba3c4"
server_key_iv: "6b2d10f0faf39533e93b3e83939f30c5"
server_key_version: "encryptionkey@0"
I20260926 01:07:18.449348 13752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } attempt: 2
I20260926 01:07:18.449765 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.449642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44629 (local address 127.0.0.1:55150)
0926 01:07:18.449699 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.449732 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 01:07:18.449771 13759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:07:18.450114 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.450018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44629 (local address 127.0.0.1:55158)
0926 01:07:18.450060 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.450086 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 01:07:18.450202 13752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44629: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44629: connect: Connection refused (error 111)
I20260926 01:07:18.450570 13774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:18.450757 13759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:18.450857 13759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/wal
uuid: "f6a66954e64b405cabdd12563463fd5f"
format_stamp: "Formatted at 2026-09-26 01:07:18 on dist-test-slave-l73v"
server_key: "e0807b149b6a79892f847f967aaba3c4"
server_key_iv: "6b2d10f0faf39533e93b3e83939f30c5"
server_key_version: "encryptionkey@0"
I20260926 01:07:18.450928 13759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:18.479509 13759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:18.481720 13759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:18.481868 13759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:18.485780 13759 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44629
I20260926 01:07:18.485836 13826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44629 every 8 connection(s)
I20260926 01:07:18.486164 13759 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
I20260926 01:07:18.487012 13827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:18.488603 13827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:18.495945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 13759
I20260926 01:07:18.496063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-1/wal/instance
I20260926 01:07:18.496248 392 external_mini_cluster.cc:1428] Setting key caaa513eb14053a305ae55bc508189ee
I20260926 01:07:18.498153 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.489407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.0.98.62:36183)
0926 01:07:18.489476 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.489478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.489633 (+ 155us) server_negotiation.cc:407] Connection header received
0926 01:07:18.490130 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.490132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.490144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.490204 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.490905 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.491437 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.492501 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.492715 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.492775 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.495039 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.495041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.495041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.495048 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.496584 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.497076 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.497077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.497078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.497141 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.497452 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.497453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.497453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.497621 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.497694 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.497954 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.498055 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 01:07:18.498250 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.489315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36183 (local address 127.0.0.1:60074)
0926 01:07:18.489517 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.489556 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:18.490020 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:18.490269 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:18.490272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:18.490601 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:18.490812 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.490817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.491555 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.491557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.492401 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.492403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.492813 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.492814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.492951 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.493453 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:18.493462 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:18.494944 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:18.496665 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.496675 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.496678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.496980 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.497205 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.497206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.497206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.497345 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.497689 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:18.497695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:18.497833 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 01:07:18.497925 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 01:07:18.498064 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260926 01:07:18.498930 13827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 01:07:18.506311 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.499207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44629 (local address 127.0.0.1:55174)
0926 01:07:18.499250 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.499260 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:18.499298 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:18.499563 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:18.499564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:18.499702 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:18.499755 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.499757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.500611 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.500612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.501153 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.501155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.501605 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.501605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.501670 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.502075 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:18.502081 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:18.503491 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:18.505299 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.505300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.505300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.505451 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.505628 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.505629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.505629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.505681 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.505954 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:18.505955 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:18.505995 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 01:07:18.506055 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 01:07:18.506166 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 01:07:18.506323 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.499248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.0.98.61:44629)
0926 01:07:18.499443 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.499447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.499461 (+ 14us) server_negotiation.cc:407] Connection header received
0926 01:07:18.499477 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.499478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.499490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.499549 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.499817 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.500538 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.501226 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.501541 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.501598 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.503602 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.503612 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.503613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.503620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.505210 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.505522 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.505526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.505526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.505575 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.505731 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.505731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.505732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.505892 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.505965 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.506097 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.506216 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260926 01:07:18.507032 13827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:07:18.507395 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.507265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51794)
0926 01:07:18.507310 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.507365 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 01:07:18.507586 13778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
I20260926 01:07:18.508109 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.507991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51808)
0926 01:07:18.508055 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.508086 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 01:07:18.508239 13827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45525: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 01:07:18.521823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45525
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36183,127.0.98.61:44629,127.0.98.60:45525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:07:18.541380 13752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } attempt: 3
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:07:18.548314 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.541872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.0.98.61:44629)
0926 01:07:18.541940 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.541941 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.541954 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:07:18.542020 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.542021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.542034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.542078 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.542436 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.542943 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.543615 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.543822 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.543879 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.545892 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.545893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.545893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.545902 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.547256 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.547588 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.547589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.547590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.547644 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.547803 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.547804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.547804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.547925 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.547983 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.548126 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.548206 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 01:07:18.548314 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.541777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44629 (local address 127.0.0.1:55182)
0926 01:07:18.541842 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.541854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:18.541896 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:18.542098 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:18.542099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:18.542254 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:18.542324 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.542326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.543025 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.543026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.543534 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.543536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.543886 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.543888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.543969 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.544406 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:18.544411 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:18.545796 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:18.547348 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.547350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.547350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.547523 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.547698 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.547698 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.547699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.547755 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.547980 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:18.547981 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:18.548036 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 01:07:18.548095 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 01:07:18.548206 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 01:07:18.548941 13752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:07:18.549306 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.549213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51812)
0926 01:07:18.549252 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.549283 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 01:07:18.549628 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.549545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51822)
0926 01:07:18.549581 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.549603 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 01:07:18.549711 13752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45525: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
I20260926 01:07:18.557476 13827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } attempt: 1
I20260926 01:07:18.558060 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.557888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51836)
0926 01:07:18.557968 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.558018 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 01:07:18.558414 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.558308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51842)
0926 01:07:18.558347 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.558393 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 01:07:18.558506 13827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45525: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
I20260926 01:07:18.580895 13752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } attempt: 1
I20260926 01:07:18.581463 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.581309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51848)
0926 01:07:18.581377 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.581431 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 01:07:18.581844 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.581752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51854)
0926 01:07:18.581796 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.581823 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:07:18.581991 13752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45525: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
W20260926 01:07:18.606808 13834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:18.607096 13834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:18.607131 13834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:18.608904 13834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:07:18.608979 13834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:18.608994 13834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:18.609005 13834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:07:18.609016 13834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:07:18.611055 13834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36183,127.0.98.61:44629,127.0.98.60:45525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:18.611327 13834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:18.611536 13834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:18.616683 13842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:18.616814 13840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:18.616696 13839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:18.616870 13834 server_base.cc:1034] running on GCE node
I20260926 01:07:18.617148 13834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:18.617427 13834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:18.618593 13834 hybrid_clock.cc:648] HybridClock initialized: now 1790384838618568 us; error 40 us; skew 500 ppm
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:18.621451 13834 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 01:07:18.621990 13834 webserver.cc:460] Webserver started at http://127.0.98.60:33767/ using document root <none> and password file <none>
I20260926 01:07:18.622212 13834 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:18.622263 13834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:18.622416 13834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:18.623610 13834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/data/instance:
uuid: "adba6bc39e714c1ab3a6d3016796e7e4"
format_stamp: "Formatted at 2026-09-26 01:07:18 on dist-test-slave-l73v"
server_key: "bf9bc298113a27589c085ea0a2323081"
server_key_iv: "acea7dd0b73e836fe032db7da243588f"
server_key_version: "encryptionkey@0"
I20260926 01:07:18.623970 13834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/wal/instance:
uuid: "adba6bc39e714c1ab3a6d3016796e7e4"
format_stamp: "Formatted at 2026-09-26 01:07:18 on dist-test-slave-l73v"
server_key: "bf9bc298113a27589c085ea0a2323081"
server_key_iv: "acea7dd0b73e836fe032db7da243588f"
server_key_version: "encryptionkey@0"
I20260926 01:07:18.625451 13834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:07:18.626343 13849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:18.626571 13834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:07:18.626624 13827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } attempt: 2
I20260926 01:07:18.626704 13834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/wal
uuid: "adba6bc39e714c1ab3a6d3016796e7e4"
format_stamp: "Formatted at 2026-09-26 01:07:18 on dist-test-slave-l73v"
server_key: "bf9bc298113a27589c085ea0a2323081"
server_key_iv: "acea7dd0b73e836fe032db7da243588f"
server_key_version: "encryptionkey@0"
I20260926 01:07:18.626845 13834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:18.627007 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.626894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51866)
0926 01:07:18.626935 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.626972 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 01:07:18.627286 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.627205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51872)
0926 01:07:18.627241 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.627264 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 01:07:18.627359 13827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45525: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45525: connect: Connection refused (error 111)
I20260926 01:07:18.649176 13834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:18.651497 13834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:18.651648 13834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:18.655793 13834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45525
I20260926 01:07:18.655822 13901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45525 every 8 connection(s)
I20260926 01:07:18.656165 13834 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
I20260926 01:07:18.657060 13902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:18.657213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 13834
I20260926 01:07:18.657174 13752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } attempt: 2
I20260926 01:07:18.657325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790384680280236-392-0/minicluster-data/master-2/wal/instance
I20260926 01:07:18.657542 392 external_mini_cluster.cc:1428] Setting key 95b1e8b23b100d72b622748a88181aab
I20260926 01:07:18.659505 13902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:18.667500 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.657493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51878)
0926 01:07:18.657551 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.657562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:18.657607 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:18.658937 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:18.658939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:18.659105 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:18.659150 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.659152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.660568 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.660569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.661084 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.661086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.661712 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.661713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.661773 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.662256 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:18.662261 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:18.664215 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:18.666106 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.666107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.666108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.666321 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.666600 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.666600 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.666601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.666666 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.667109 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:18.667110 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:18.667161 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 01:07:18.667288 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 01:07:18.667398 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 01:07:18.668120 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.660458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.0.98.62:36183)
0926 01:07:18.660496 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.660498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.660508 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:07:18.660534 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.660535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.660544 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.660607 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.661205 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.661786 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.662782 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.663077 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.663154 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.665449 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.665451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.665451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.665458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.666934 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.667314 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.667315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.667315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.667367 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.667567 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.667568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.667568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.667693 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.667734 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.667964 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.668035 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:07:18.668205 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.658019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.0.98.60:45525)
0926 01:07:18.658124 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.658126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.658133 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:07:18.658690 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.658702 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.658756 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.658903 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.659786 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.660529 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.661527 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.661784 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.661838 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.664426 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.664428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.664428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.664435 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.666775 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.667172 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.667173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.667173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.667224 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.667393 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.667394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.667394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.667497 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.667770 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.667808 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.667968 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":88,"thread_start_us":67,"threads_started":1}
I20260926 01:07:18.668210 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.660200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36183 (local address 127.0.0.1:60086)
0926 01:07:18.660387 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.660428 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:18.660489 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:18.660673 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:18.660676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:18.661043 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:18.661112 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.661114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.661883 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.661884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.662681 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.662684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.663197 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.663199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.663282 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.663719 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:18.663730 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:18.665379 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:18.666992 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.667000 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.667002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.667224 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.667436 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.667436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.667437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.667503 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.667776 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:18.667781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:18.667845 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 01:07:18.667924 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 01:07:18.668030 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260926 01:07:18.668424 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.657648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.0.98.60:45525)
0926 01:07:18.657881 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.657890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.657917 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:07:18.658685 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.658695 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.658756 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.658902 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.659251 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.660447 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.661162 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.661635 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.661745 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.664315 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.664324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.664324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.664331 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.666028 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.666415 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.666418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.666419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.666515 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.666720 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.666721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.666721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.667039 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.667124 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.667776 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.668341 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":411,"server-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
I20260926 01:07:18.668807 13902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:07:18.670055 13752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d: Bootstrap starting.
I20260926 01:07:18.670800 13752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:18.671118 13752 log.cc:824] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:18.672248 13752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d: No bootstrap required, opened a new log
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:07:18.674201 13752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } }
I20260926 01:07:18.674319 13752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:18.674340 13752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b7a62688d743ff85c0396323b2d15d, State: Initialized, Role: FOLLOWER
I20260926 01:07:18.674471 13752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } }
I20260926 01:07:18.674811 13752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:18.675307 13909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } } }
I20260926 01:07:18.675390 13909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:07:18.675518 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.669120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.0.98.61:44629)
0926 01:07:18.669163 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.669165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.669199 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:07:18.669258 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.669259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.669268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.669319 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.669572 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.670040 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.670825 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.671066 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.671125 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.672868 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.672869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.672870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.672878 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.674478 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.674796 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.674797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.674797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.674855 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.675022 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.675023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.675023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.675132 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.675186 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.675343 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.675424 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:07:18.675787 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.669059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44629 (local address 127.0.0.1:55194)
0926 01:07:18.669159 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.669170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:18.669212 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:18.669323 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:18.669324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:18.669455 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:18.669505 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.669507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.670110 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.670111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.670763 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.670766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.671130 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.671131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.671201 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.671622 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:18.671626 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:18.672787 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:18.674572 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.674573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.674574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.674728 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.674909 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.674909 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.674910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.674972 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.675183 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:18.675184 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:18.675228 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:07:18.675619 (+ 391us) client_negotiation.cc:240] Negotiation successful
0926 01:07:18.675713 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 01:07:18.676101 13902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } has no permanent_uuid. Determining permanent_uuid...
W20260926 01:07:18.676910 13920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:07:18.677016 13920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:07:18.678491 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.670688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.0.98.62:36183)
0926 01:07:18.670723 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.670724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.670733 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:07:18.670803 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.670804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.670814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.670861 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.671450 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.671936 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.672885 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.673162 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.673222 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.675612 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.675614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.675614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.675621 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.677237 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.677629 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.677630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.677631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.677684 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.677869 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.677870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.677870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.677991 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.678093 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.678289 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.678411 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:07:18.677729 13684 master_runner.cc:186] Error getting master registration for 127.0.98.61:44629: OK, instance_id { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" instance_seqno: 1790384838484616 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:07:18.680547 13684 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:07:18.684643 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.676594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.0.98.60:45525)
0926 01:07:18.676630 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.676631 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.676793 (+ 162us) server_negotiation.cc:407] Connection header received
0926 01:07:18.676809 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.676810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.676818 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.676854 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.677164 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.677837 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.678798 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.679078 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.679132 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.681235 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.681237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.681237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.681246 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.682914 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.683320 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.683321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.683321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.683388 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.684064 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.684066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.684066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.684252 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.684323 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.684451 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.684545 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":18}
I20260926 01:07:18.685030 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.676361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51902)
0926 01:07:18.676695 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.676705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:18.676748 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:18.676884 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:18.676885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:18.677046 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:18.677087 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.677089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.677911 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.677912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.678709 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.678713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.679180 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.679181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.679224 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.679609 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:18.679615 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:18.681108 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:18.683011 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.683012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.683013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.683236 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.683909 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.683910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.683910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.683981 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.684318 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:18.684319 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:18.684362 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 01:07:18.684426 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:07:18.684938 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":403}
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:07:18.686842 13902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4: Bootstrap starting.
I20260926 01:07:18.687685 13902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:18.688014 13902 log.cc:824] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:18.688941 13902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4: No bootstrap required, opened a new log
I20260926 01:07:18.689079 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.680636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.0.98.61:44629)
0926 01:07:18.680677 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.680679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.680687 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:07:18.680710 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.680711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.680719 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.680772 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.681853 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.682632 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.683476 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.683771 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.683846 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.686608 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.686609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.686610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.686618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.687978 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.688333 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.688334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.688334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.688401 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.688597 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.688598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.688598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.688712 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.688880 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.688916 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.688990 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:07:18.690374 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:07:18.691550 13902 raft_consensus.cc:348] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } }
I20260926 01:07:18.691736 13902 raft_consensus.cc:374] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:18.691866 13902 raft_consensus.cc:729] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adba6bc39e714c1ab3a6d3016796e7e4, State: Initialized, Role: FOLLOWER
I20260926 01:07:18.692091 13902 consensus_queue.cc:260] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } }
I20260926 01:07:18.692538 13902 sys_catalog.cc:564] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:07:18.694017 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } } }
I20260926 01:07:18.694182 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:07:18.694787 13941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:07:18.694900 13941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:07:18.696288 13834 master_runner.cc:186] Error getting master registration for 127.0.98.61:44629: OK, instance_id { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" instance_seqno: 1790384838484616 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:07:18.698619 13834 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:07:18.699689 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.691692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.0.98.62:36183)
0926 01:07:18.691841 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.691843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.691974 (+ 131us) server_negotiation.cc:407] Connection header received
0926 01:07:18.692079 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.692080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.692092 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.692122 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.692789 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.693381 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.694328 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.694641 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.694707 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.696208 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.696209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.696210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.696217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.697825 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.698212 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.698213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.698213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.698286 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.698832 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.698833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.698834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.698974 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.699065 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.699511 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.699593 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 01:07:18.708263 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.692167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.0.98.60:45525)
0926 01:07:18.692216 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.692217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.695270 (+ 3053us) server_negotiation.cc:407] Connection header received
0926 01:07:18.695374 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.695375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.695386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.695413 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.696077 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.696736 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.697696 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.697959 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.698003 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.705064 (+ 7061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.705066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.705067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.705077 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.706833 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.707265 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.707267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.707267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.707328 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.707553 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.707554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.707555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.707726 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.707845 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.708072 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.708160 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:07:18.708416 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.691952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.0.98.61:44629)
0926 01:07:18.691986 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.691988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.695109 (+ 3121us) server_negotiation.cc:407] Connection header received
0926 01:07:18.695176 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.695177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.695190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.695238 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.700382 (+ 5144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.701125 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.702167 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.702502 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.702552 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.705065 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.705066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.705067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.705076 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.707157 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.707559 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.707560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.707561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.707619 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.707809 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.707810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.707810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.707946 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.708027 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.708225 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.708317 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:07:18.718497 13827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } attempt: 3
Sep 26 01:07:18 dist-test-slave-l73v krb5kdc[13671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:07:18.729508 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.718864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45525 (local address 127.0.0.1:51910)
0926 01:07:18.718924 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:18.718935 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:18.718978 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:18.719572 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:18.719573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:18.719777 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:18.719832 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.719834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.720856 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.720857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.721679 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:18.721683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.722171 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.722172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:18.722236 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.722897 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:18.722903 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:18.725045 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:18.728114 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.728115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.728116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.728411 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.728734 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:18.728736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:18.728736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:18.728828 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:18.729141 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:18.729142 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:18.729197 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 01:07:18.729269 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 01:07:18.729387 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 01:07:18.729516 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:18.719367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.0.98.60:45525)
0926 01:07:18.719411 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:18.719413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:18.719421 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:07:18.719445 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:18.719446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:18.719455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:18.719518 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:18.719950 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.720743 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.721774 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:18.722090 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:18.722166 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:18.725159 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:18.725161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:18.725161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:18.725169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:18.727972 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.728572 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.728574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.728574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.728639 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:18.728916 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:18.728916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:18.728917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:18.729070 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:18.729141 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:18.729318 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 01:07:18.729405 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:07:18.731786 13827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f: Bootstrap starting.
I20260926 01:07:18.732646 13827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:18.733086 13827 log.cc:824] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:18.734194 13827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f: No bootstrap required, opened a new log
I20260926 01:07:18.736281 13827 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } }
I20260926 01:07:18.736464 13827 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:18.736502 13827 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a66954e64b405cabdd12563463fd5f, State: Initialized, Role: FOLLOWER
I20260926 01:07:18.736631 13827 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } }
I20260926 01:07:18.736986 13827 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:07:18.736939 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } } }
I20260926 01:07:18.737187 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:07:18.739902 13955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:07:18.740041 13955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:07:18.741052 13759 master_runner.cc:263] No leader master found from master f6a66954e64b405cabdd12563463fd5f
I20260926 01:07:18.741104 13759 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:07:18.812124 13705 tablet.cc:2369] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:07:18.820587 13929 raft_consensus.cc:482] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:07:18.820734 13929 raft_consensus.cc:504] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } }
I20260926 01:07:18.820992 13929 leader_election.cc:288] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b7a62688d743ff85c0396323b2d15d (127.0.98.62:36183), f6a66954e64b405cabdd12563463fd5f (127.0.98.61:44629)
I20260926 01:07:18.821451 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a66954e64b405cabdd12563463fd5f" is_pre_election: true
I20260926 01:07:18.821645 13801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adba6bc39e714c1ab3a6d3016796e7e4 in term 0.
I20260926 01:07:18.822149 13726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b7a62688d743ff85c0396323b2d15d" is_pre_election: true
I20260926 01:07:18.822853 13850 leader_election.cc:302] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adba6bc39e714c1ab3a6d3016796e7e4, f6a66954e64b405cabdd12563463fd5f; no voters:
I20260926 01:07:18.822365 13726 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adba6bc39e714c1ab3a6d3016796e7e4 in term 0.
I20260926 01:07:18.823292 13929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:07:18.823757 13929 raft_consensus.cc:482] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:07:18.823796 13929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:18.824760 13929 raft_consensus.cc:504] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } }
I20260926 01:07:18.824900 13929 leader_election.cc:288] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [CANDIDATE]: Term 1 election: Requested vote from peers 79b7a62688d743ff85c0396323b2d15d (127.0.98.62:36183), f6a66954e64b405cabdd12563463fd5f (127.0.98.61:44629)
I20260926 01:07:18.825352 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a66954e64b405cabdd12563463fd5f"
I20260926 01:07:18.825446 13801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:18.826995 13801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adba6bc39e714c1ab3a6d3016796e7e4 in term 1.
I20260926 01:07:18.827303 13850 leader_election.cc:302] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adba6bc39e714c1ab3a6d3016796e7e4, f6a66954e64b405cabdd12563463fd5f; no voters:
I20260926 01:07:18.827494 13726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b7a62688d743ff85c0396323b2d15d"
I20260926 01:07:18.827587 13726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:18.828588 13726 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adba6bc39e714c1ab3a6d3016796e7e4 in term 1.
I20260926 01:07:18.828940 13929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:07:18.829103 13929 raft_consensus.cc:686] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [term 1 LEADER]: Becoming Leader. State: Replica: adba6bc39e714c1ab3a6d3016796e7e4, State: Running, Role: LEADER
I20260926 01:07:18.829197 13929 consensus_queue.cc:237] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } }
I20260926 01:07:18.829639 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adba6bc39e714c1ab3a6d3016796e7e4. Latest consensus state: current_term: 1 leader_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } } }
I20260926 01:07:18.829715 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:18.831315 13959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:07:18.831600 13959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:07:18.833976 13801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [term 1 FOLLOWER]: Refusing update from remote peer adba6bc39e714c1ab3a6d3016796e7e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:07:18.834478 13929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:07:18.836282 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader adba6bc39e714c1ab3a6d3016796e7e4. Latest consensus state: current_term: 1 leader_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } } }
I20260926 01:07:18.836398 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:07:18.837060 13726 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [term 1 FOLLOWER]: Refusing update from remote peer adba6bc39e714c1ab3a6d3016796e7e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:07:18.837486 13929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:07:18.838904 13909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [sys.catalog]: SysCatalogTable state changed. Reason: New leader adba6bc39e714c1ab3a6d3016796e7e4. Latest consensus state: current_term: 1 leader_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } } }
I20260926 01:07:18.839005 13909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:07:18.840498 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } } }
I20260926 01:07:18.841084 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } } }
I20260926 01:07:18.841169 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:07:18.841463 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:18.841598 13960 mvcc.cc:205] Tried to move back new op lower bound from 7333416299861151744 to 7333416299844939776. Current Snapshot: MvccSnapshot[applied={T|T < 7333416299861151744}]
I20260926 01:07:18.841711 13961 mvcc.cc:205] Tried to move back new op lower bound from 7333416299861151744 to 7333416299844939776. Current Snapshot: MvccSnapshot[applied={T|T < 7333416299861151744}]
I20260926 01:07:18.842973 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } } }
I20260926 01:07:18.843062 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:18.843649 13909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b7a62688d743ff85c0396323b2d15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36183 } } peers { permanent_uuid: "f6a66954e64b405cabdd12563463fd5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44629 } } peers { permanent_uuid: "adba6bc39e714c1ab3a6d3016796e7e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45525 } } }
I20260926 01:07:18.843739 13909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:07:18.843987 13963 mvcc.cc:205] Tried to move back new op lower bound from 7333416299861151744 to 7333416299844939776. Current Snapshot: MvccSnapshot[applied={T|T < 7333416299861151744}]
I20260926 01:07:18.845043 13959 catalog_manager.cc:1347] Generated new cluster ID: 70039534a6944287880ead348da138c8
I20260926 01:07:18.845093 13959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:07:18.864013 13959 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:07:18.864809 13959 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:07:18.875835 13959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adba6bc39e714c1ab3a6d3016796e7e4: Generated new TSK 0
I20260926 01:07:18.876093 13959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:07:19.677536 13920 catalog_manager.cc:1259] Loaded cluster ID: 70039534a6944287880ead348da138c8
I20260926 01:07:19.677626 13920 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d: loading cluster ID for follower catalog manager: success
I20260926 01:07:19.678443 13920 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d: acquiring CA information for follower catalog manager: success
I20260926 01:07:19.678722 13920 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79b7a62688d743ff85c0396323b2d15d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:07:19.740566 13955 catalog_manager.cc:1259] Loaded cluster ID: 70039534a6944287880ead348da138c8
I20260926 01:07:19.740654 13955 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f: loading cluster ID for follower catalog manager: success
I20260926 01:07:19.741451 13955 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f: acquiring CA information for follower catalog manager: success
I20260926 01:07:19.741721 13955 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6a66954e64b405cabdd12563463fd5f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:07:19.931805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 13684
I20260926 01:07:19.936494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 13759
I20260926 01:07:19.940799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 13834
2026-09-26T01:07:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2849 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:07:20 dist-test-slave-l73v krb5kdc[13974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:07:20 dist-test-slave-l73v krb5kdc[13974](info): set up 2 sockets
Sep 26 01:07:20 dist-test-slave-l73v krb5kdc[13974](info): commencing operation
krb5kdc: starting...
W20260926 01:07:20.910615 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:07:20 dist-test-slave-l73v krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:07:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:07:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:07:21.034618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44173
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:21.118242 13987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:21.118520 13987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:21.118546 13987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:21.119926 13987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:07:21.119987 13987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:21.120000 13987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:21.120012 13987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:07:21.120023 13987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:07:21.121737 13987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:21.122022 13987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:21.122287 13987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:21.127477 13992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:21.127486 13995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:21.127496 13993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:21.127789 13987 server_base.cc:1034] running on GCE node
I20260926 01:07:21.128036 13987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:21.128499 13987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:21.129738 13987 hybrid_clock.cc:648] HybridClock initialized: now 1790384841129722 us; error 31 us; skew 500 ppm
Sep 26 01:07:21 dist-test-slave-l73v krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:21.132629 13987 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:07:21.133129 13987 webserver.cc:460] Webserver started at http://127.0.98.62:35465/ using document root <none> and password file <none>
I20260926 01:07:21.133351 13987 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:21.133396 13987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:21.133525 13987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:21.134663 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0"
format_stamp: "Formatted at 2026-09-26 01:07:21 on dist-test-slave-l73v"
server_key: "97a28298fc453bd93c7f873b0e231c47"
server_key_iv: "66eab33ad2329bc6741bccaf3226698e"
server_key_version: "encryptionkey@0"
I20260926 01:07:21.135010 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0"
format_stamp: "Formatted at 2026-09-26 01:07:21 on dist-test-slave-l73v"
server_key: "97a28298fc453bd93c7f873b0e231c47"
server_key_iv: "66eab33ad2329bc6741bccaf3226698e"
server_key_version: "encryptionkey@0"
I20260926 01:07:21.136400 13987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:21.137302 14002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:21.137615 13987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:21.137718 13987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0"
format_stamp: "Formatted at 2026-09-26 01:07:21 on dist-test-slave-l73v"
server_key: "97a28298fc453bd93c7f873b0e231c47"
server_key_iv: "66eab33ad2329bc6741bccaf3226698e"
server_key_version: "encryptionkey@0"
I20260926 01:07:21.137809 13987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:21.155238 13987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:21.157534 13987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:21.157663 13987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:21.161785 13987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44173
I20260926 01:07:21.161909 14054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44173 every 8 connection(s)
I20260926 01:07:21.162175 13987 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:07:21.163065 14055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:21.165395 14055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0: Bootstrap starting.
I20260926 01:07:21.165935 14055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:21.166165 14055 log.cc:824] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:21.166917 14055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0: No bootstrap required, opened a new log
I20260926 01:07:21.168543 14055 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44173 } }
I20260926 01:07:21.168696 14055 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:21.168720 14055 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0e4cf5f5374b2abc5e957a3bd6d4a0, State: Initialized, Role: FOLLOWER
I20260926 01:07:21.168836 14055 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44173 } }
I20260926 01:07:21.168912 14055 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:07:21.168937 14055 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:07:21.168965 14055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:21.169526 14055 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44173 } }
I20260926 01:07:21.169613 14055 leader_election.cc:302] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd0e4cf5f5374b2abc5e957a3bd6d4a0; no voters:
I20260926 01:07:21.169749 14055 leader_election.cc:288] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:07:21.169744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 13987
I20260926 01:07:21.169827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:07:21.169829 14058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:07:21.169941 14058 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [term 1 LEADER]: Becoming Leader. State: Replica: dd0e4cf5f5374b2abc5e957a3bd6d4a0, State: Running, Role: LEADER
I20260926 01:07:21.170024 392 external_mini_cluster.cc:1428] Setting key bd88a8b2d66f11f31655ad112409366d
I20260926 01:07:21.170020 14055 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:07:21.170012 14058 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44173 } }
I20260926 01:07:21.170445 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd0e4cf5f5374b2abc5e957a3bd6d4a0. Latest consensus state: current_term: 1 leader_uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44173 } } }
I20260926 01:07:21.170632 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:21.170655 14059 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e4cf5f5374b2abc5e957a3bd6d4a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44173 } } }
I20260926 01:07:21.170742 14059 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0 [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:21.170890 14069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:07:21.171448 14069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:07:21.173089 14069 catalog_manager.cc:1347] Generated new cluster ID: e3547887820c4b9e88496a5f4d0c384b
I20260926 01:07:21.173132 14069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:07:21 dist-test-slave-l73v krb5kdc[13974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:21.183382 14069 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:07:21.183544 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:21.170668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.0.98.62:44173)
0926 01:07:21.170845 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:21.170856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:21.171192 (+ 336us) server_negotiation.cc:407] Connection header received
0926 01:07:21.171437 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:21.171446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:21.171509 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:21.171629 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:21.172352 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:21.173704 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:21.174818 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:21.175487 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:21.175611 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:21.179102 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:21.179109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:21.179109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:21.179126 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:21.181427 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:21.181971 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:21.181975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:21.181975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:21.182089 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:21.182312 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:21.182313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:21.182313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:21.182757 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:21.183006 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:21.183090 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 01:07:21.183236 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260926 01:07:21.184239 14069 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:07:21.184508 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:07:21.202129 14069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd0e4cf5f5374b2abc5e957a3bd6d4a0: Generated new TSK 0
I20260926 01:07:21.202353 14069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:07:21.238287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 13987
2026-09-26T01:07:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1295 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:07:21 dist-test-slave-l73v krb5kdc[14082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:07:21 dist-test-slave-l73v krb5kdc[14082](info): set up 2 sockets
Sep 26 01:07:21 dist-test-slave-l73v krb5kdc[14082](info): commencing operation
krb5kdc: starting...
W20260926 01:07:22.194229 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:07:22 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:07:22.313988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45651
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:22.395576 14095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:22.395807 14095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:22.395833 14095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:22.397096 14095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:07:22.397151 14095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:22.397166 14095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:22.397176 14095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:07:22.397187 14095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:07:22.398808 14095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:22.399066 14095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:22.399288 14095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:22.404516 14103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:22.404491 14100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:22.404531 14095 server_base.cc:1034] running on GCE node
W20260926 01:07:22.404495 14101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:22.404999 14095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:22.405354 14095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:22.406510 14095 hybrid_clock.cc:648] HybridClock initialized: now 1790384842406490 us; error 31 us; skew 500 ppm
Sep 26 01:07:22 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:22.409200 14095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:07:22.409730 14095 webserver.cc:460] Webserver started at http://127.0.98.62:38959/ using document root <none> and password file <none>
I20260926 01:07:22.409940 14095 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:22.409991 14095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:22.410120 14095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:22.411285 14095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "4e76b9b16b835a7c01dc9959e374b1a4"
server_key_iv: "c6172454d04c7d50281fff6222895f83"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.411655 14095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "4e76b9b16b835a7c01dc9959e374b1a4"
server_key_iv: "c6172454d04c7d50281fff6222895f83"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.413092 14095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:07:22.414057 14110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:22.414472 14095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:07:22.414565 14095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "4e76b9b16b835a7c01dc9959e374b1a4"
server_key_iv: "c6172454d04c7d50281fff6222895f83"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.414645 14095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:22.426437 14095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:22.428705 14095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:22.428853 14095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:22.432778 14095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45651
I20260926 01:07:22.432821 14162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45651 every 8 connection(s)
I20260926 01:07:22.433171 14095 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:07:22.434108 14163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:22.436477 14163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Bootstrap starting.
I20260926 01:07:22.437038 14163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:22.437290 14163 log.cc:824] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:22.438032 14163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: No bootstrap required, opened a new log
I20260926 01:07:22.438935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 14095
I20260926 01:07:22.439047 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:07:22.439234 392 external_mini_cluster.cc:1428] Setting key 645c939b41a970562bf6b373c95e9b8e
I20260926 01:07:22.440158 14163 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:22.440311 14163 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:22.440371 14163 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf4c5aa7a3249999bbf8963ac7e3fff, State: Initialized, Role: FOLLOWER
I20260926 01:07:22.440521 14163 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:22.440585 14163 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:07:22.440606 14163 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:07:22.440650 14163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:22.441550 14163 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:22.441649 14163 leader_election.cc:302] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf4c5aa7a3249999bbf8963ac7e3fff; no voters:
I20260926 01:07:22.441802 14163 leader_election.cc:288] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:07:22.441843 14168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:07:22.441992 14168 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 1 LEADER]: Becoming Leader. State: Replica: cbf4c5aa7a3249999bbf8963ac7e3fff, State: Running, Role: LEADER
I20260926 01:07:22.442060 14163 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:07:22.442114 14168 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:22.442451 14168 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf4c5aa7a3249999bbf8963ac7e3fff. Latest consensus state: current_term: 1 leader_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } } }
I20260926 01:07:22.442517 14168 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:22.442710 14174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:07:22.443383 14174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:07:22.443547 14169 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } } }
I20260926 01:07:22.443727 14169 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:22.445027 14174 catalog_manager.cc:1347] Generated new cluster ID: 888096b31e384ae18d8a3baa697b3d41
I20260926 01:07:22.445078 14174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:07:22 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:22.450515 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.439826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.0.98.62:45651)
0926 01:07:22.440013 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:22.440021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:22.440047 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:07:22.440148 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:22.440154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:22.440199 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:22.440316 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:22.441050 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.442080 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.443233 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.443814 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.443935 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.446757 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:22.446762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:22.446762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:22.446776 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:22.448744 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:22.449164 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:22.449168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:22.449169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:22.449317 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:22.449489 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:22.449490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:22.449490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:22.449835 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:22.450011 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:22.450084 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 01:07:22.450236 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260926 01:07:22.458493 14174 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:07:22.459051 14174 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:07:22.465168 14174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Generated new TSK 0
I20260926 01:07:22.465358 14174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:07:22.498297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:22.581514 14189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:22.581737 14189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:22.581763 14189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:22.583029 14189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:22.583081 14189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:22.583108 14189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:07:22.584643 14189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:22.584899 14189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:22.585139 14189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:22.590132 14195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:22.590121 14197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:22.590121 14194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:22.590227 14189 server_base.cc:1034] running on GCE node
I20260926 01:07:22.590623 14189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:22.590884 14189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:22.592057 14189 hybrid_clock.cc:648] HybridClock initialized: now 1790384842592045 us; error 30 us; skew 500 ppm
Sep 26 01:07:22 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:22.594695 14189 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:07:22.595185 14189 webserver.cc:460] Webserver started at http://127.0.98.1:35109/ using document root <none> and password file <none>
I20260926 01:07:22.595394 14189 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:22.595438 14189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:22.595561 14189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:22.596622 14189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "0431139c48aa4ca88c3246a73f228cce"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "d89aa8acd101fe6ee5623e9bc7acb4f9"
server_key_iv: "2417e693987c51bf43ff6bd9a55750af"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.596964 14189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0431139c48aa4ca88c3246a73f228cce"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "d89aa8acd101fe6ee5623e9bc7acb4f9"
server_key_iv: "2417e693987c51bf43ff6bd9a55750af"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.598356 14189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 01:07:22.599255 14204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:22.599443 14189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:22.599525 14189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "0431139c48aa4ca88c3246a73f228cce"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "d89aa8acd101fe6ee5623e9bc7acb4f9"
server_key_iv: "2417e693987c51bf43ff6bd9a55750af"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.599608 14189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:22.611428 14189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:22.613848 14189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:22.613983 14189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:22.614220 14189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:22.614574 14189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:22.614610 14189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:22.614640 14189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:22.614666 14189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:22.620208 14189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38111
I20260926 01:07:22.620265 14317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38111 every 8 connection(s)
I20260926 01:07:22.620582 14189 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:07:22.623829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 14189
I20260926 01:07:22.623926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:07:22.624110 392 external_mini_cluster.cc:1428] Setting key f2b08286fb2bd444cf4814b1ed869ed3
Sep 26 01:07:22 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:22.631269 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.621349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45691 (local address 127.0.98.62:45651)
0926 01:07:22.621439 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:22.621441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:22.621580 (+ 139us) server_negotiation.cc:407] Connection header received
0926 01:07:22.622049 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:22.622050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:22.622067 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:22.622142 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:22.623135 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.623687 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.625052 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.625268 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.625326 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.628051 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:22.628054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:22.628054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:22.628066 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:22.629547 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:22.630108 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:22.630109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:22.630110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:22.630178 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:22.630510 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:22.630511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:22.630511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:22.630644 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:22.630707 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:22.631105 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 01:07:22.631196 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 01:07:22.631466 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.621254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:45691)
0926 01:07:22.621417 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:22.621462 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:22.621936 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:22.622198 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:22.622202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:22.622814 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:22.623018 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.623023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.623807 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.623809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.624969 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.624972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.625345 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.625346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.625492 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.626150 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:22.626162 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:22.627913 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:22.629650 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:22.629662 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:22.629665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:22.630016 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:22.630253 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:22.630254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:22.630254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:22.630444 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:22.630716 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:22.630723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:22.630918 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 01:07:22.631035 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 01:07:22.631219 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:07:22.632144 14318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45651
I20260926 01:07:22.632244 14318 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:22.632556 14318 heartbeater.cc:511] Master 127.0.98.62:45651 requested a full tablet report, sending...
I20260926 01:07:22.633239 14127 ts_manager.cc:194] Registered new tserver with Master: 0431139c48aa4ca88c3246a73f228cce (127.0.98.1:38111)
I20260926 01:07:22.633886 14127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:07:22.653196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:22.736275 14323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:22.736511 14323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:22.736536 14323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:22.737788 14323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:22.737846 14323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:22.737876 14323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:07:22.739506 14323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:22.739734 14323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:22.739939 14323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:22.745868 14329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:22.745975 14328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:22.745875 14331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:22.746140 14323 server_base.cc:1034] running on GCE node
I20260926 01:07:22.746465 14323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:22.746748 14323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:22.747949 14323 hybrid_clock.cc:648] HybridClock initialized: now 1790384842747921 us; error 41 us; skew 500 ppm
Sep 26 01:07:22 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:22.751519 14323 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:07:22.752059 14323 webserver.cc:460] Webserver started at http://127.0.98.2:32829/ using document root <none> and password file <none>
I20260926 01:07:22.752285 14323 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:22.752338 14323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:22.752481 14323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:22.753702 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "183c053291a0441b9b59751807b2d040"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "1a5ab8d2e1285482f88fef5a07bcf396"
server_key_iv: "0123e5e343770060f907214c38f49efb"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.754146 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "183c053291a0441b9b59751807b2d040"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "1a5ab8d2e1285482f88fef5a07bcf396"
server_key_iv: "0123e5e343770060f907214c38f49efb"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.756033 14323 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260926 01:07:22.756935 14338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:22.757086 14323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:07:22.757181 14323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "183c053291a0441b9b59751807b2d040"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "1a5ab8d2e1285482f88fef5a07bcf396"
server_key_iv: "0123e5e343770060f907214c38f49efb"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.757261 14323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:22.778801 14323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:22.780927 14323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:22.781083 14323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:22.781312 14323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:22.781615 14323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:22.781648 14323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:22.781677 14323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:22.781694 14323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:22.787189 14323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41135
I20260926 01:07:22.787241 14451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41135 every 8 connection(s)
I20260926 01:07:22.787559 14323 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:07:22.788561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 14323
I20260926 01:07:22.788666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:07:22.788843 392 external_mini_cluster.cc:1428] Setting key 307092f8cb027ea8d2a5c5702d96d9bc
Sep 26 01:07:22 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:07:22.798130 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.788255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53343 (local address 127.0.98.62:45651)
0926 01:07:22.788354 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:22.788356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:22.788578 (+ 222us) server_negotiation.cc:407] Connection header received
0926 01:07:22.789227 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:22.789229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:22.789241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:22.789291 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:22.790325 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.791071 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.792146 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.792425 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.792503 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.795045 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:22.795047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:22.795048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:22.795058 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:22.796653 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:22.797157 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:22.797159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:22.797159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:22.797221 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:22.797470 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:22.797471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:22.797472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:22.797594 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:22.797653 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:22.797975 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 01:07:22.798058 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 01:07:22.798291 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.788219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:53343)
0926 01:07:22.788448 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:22.788488 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:22.789115 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:22.789371 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:22.789375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:22.790020 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:22.790223 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.790228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.791207 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.791208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.792057 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.792059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.792582 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.792583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.792705 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.793263 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:22.793272 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:22.794929 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:22.796753 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:22.796763 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:22.796765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:22.797080 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:22.797286 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:22.797287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:22.797287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:22.797408 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:22.797653 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:22.797661 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:22.797810 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 01:07:22.797917 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 01:07:22.798080 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260926 01:07:22.798833 14452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45651
I20260926 01:07:22.798974 14452 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:22.799160 14452 heartbeater.cc:511] Master 127.0.98.62:45651 requested a full tablet report, sending...
I20260926 01:07:22.799582 14127 ts_manager.cc:194] Registered new tserver with Master: 183c053291a0441b9b59751807b2d040 (127.0.98.2:41135)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 01:07:22.799980 14127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:07:22.821933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:22.906807 14457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:22.907051 14457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:22.907083 14457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:22.908648 14457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:22.908720 14457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:22.908752 14457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:07:22.910522 14457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:22.910770 14457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:22.910975 14457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:22.916324 14465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:22.916330 14462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:22.916318 14463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:22.916575 14457 server_base.cc:1034] running on GCE node
I20260926 01:07:22.916893 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:22.917162 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:22.918346 14457 hybrid_clock.cc:648] HybridClock initialized: now 1790384842918326 us; error 29 us; skew 500 ppm
Sep 26 01:07:22 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:22.921407 14457 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:07:22.921944 14457 webserver.cc:460] Webserver started at http://127.0.98.3:39855/ using document root <none> and password file <none>
I20260926 01:07:22.922169 14457 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:22.922221 14457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:22.922349 14457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:22.923489 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "8353cc86dfca458294fac6c8c1ccba52"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "bd91811126faa0c48206a7f097234c76"
server_key_iv: "a03d3b312e721bf15e482ee454bc6f50"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.923857 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8353cc86dfca458294fac6c8c1ccba52"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "bd91811126faa0c48206a7f097234c76"
server_key_iv: "a03d3b312e721bf15e482ee454bc6f50"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.925285 14457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:07:22.926178 14472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:22.926447 14457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:22.926582 14457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "8353cc86dfca458294fac6c8c1ccba52"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "bd91811126faa0c48206a7f097234c76"
server_key_iv: "a03d3b312e721bf15e482ee454bc6f50"
server_key_version: "encryptionkey@0"
I20260926 01:07:22.926666 14457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:22.938601 14457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:22.941663 14457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:22.941946 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:22.942233 14457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:22.942687 14457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:22.942732 14457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:22.942763 14457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:22.942780 14457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:22.949620 14457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35427
I20260926 01:07:22.949666 14585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35427 every 8 connection(s)
I20260926 01:07:22.949994 14457 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:07:22 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:22.957342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 14457
I20260926 01:07:22.957444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:07:22.957628 392 external_mini_cluster.cc:1428] Setting key 97bbab3b0cd08aeea82c8ddabd09665c
I20260926 01:07:22.959880 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.950768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50029 (local address 127.0.98.62:45651)
0926 01:07:22.950819 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:22.950821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:22.951039 (+ 218us) server_negotiation.cc:407] Connection header received
0926 01:07:22.951633 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:22.951634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:22.951647 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:22.951722 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:22.952832 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.953379 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.954466 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.954661 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.954717 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.956880 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:22.956882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:22.956882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:22.956890 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:22.958641 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:22.959058 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:22.959059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:22.959059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:22.959124 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:22.959330 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:22.959331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:22.959331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:22.959437 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:22.959481 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:22.959736 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 01:07:22.959811 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:07:22.960062 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.950738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:50029)
0926 01:07:22.950913 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:22.950956 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:22.951498 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:22.951819 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:22.951823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:22.952503 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:22.952718 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.952723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.953503 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.953503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.954327 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.954329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.954728 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.954729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.954844 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.955391 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:22.955399 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:22.956773 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:22.958741 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:22.958751 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:22.958753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:22.958998 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:22.959179 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:22.959179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:22.959180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:22.959288 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:22.959472 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:22.959476 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:22.959622 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 01:07:22.959704 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 01:07:22.959837 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260926 01:07:22.960507 14586 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45651
I20260926 01:07:22.960596 14586 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:22.960777 14586 heartbeater.cc:511] Master 127.0.98.62:45651 requested a full tablet report, sending...
I20260926 01:07:22.961170 14127 ts_manager.cc:194] Registered new tserver with Master: 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427)
I20260926 01:07:22.961565 14127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50029
I20260926 01:07:22.969820 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:07:22.977422 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.971153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.0.98.62:45651)
0926 01:07:22.971227 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:22.971229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:22.971288 (+ 59us) server_negotiation.cc:407] Connection header received
0926 01:07:22.971355 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:22.971356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:22.971367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:22.971417 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:22.972024 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.972538 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.973310 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.973503 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.973560 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.974873 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:22.974875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:22.974875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:22.974884 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:22.976290 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:22.976649 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:22.976650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:22.976650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:22.976705 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:22.976868 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:22.976868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:22.976869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:22.977029 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:22.977120 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:22.977241 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 01:07:22.977325 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 01:07:22.979387 14127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:07:22.981027 14127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:07:22.988111 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.983997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.0.98.1:38111)
0926 01:07:22.984142 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:22.984148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:22.984488 (+ 340us) server_negotiation.cc:407] Connection header received
0926 01:07:22.984542 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:22.984544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:22.984561 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:22.984629 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:22.984915 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.986016 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.987184 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.987705 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.987770 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.987838 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:22.987972 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 01:07:22.988017 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260926 01:07:22.988104 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.983821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.0.1:47126)
0926 01:07:22.984337 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:22.984348 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:22.984414 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:22.984772 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:22.984774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:22.984776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:22.984828 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.984829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.986117 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.986118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.987110 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.987112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.987784 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.987785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.987867 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.987882 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:07:22.987956 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 01:07:22.988016 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":82,"threads_started":1}
I20260926 01:07:22.988253 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.984034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35427 (local address 127.0.0.1:51508)
0926 01:07:22.984421 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:22.984429 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:22.984473 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:22.984728 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:22.984729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:22.984730 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:22.984768 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.984769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.985680 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.985682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.986936 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.986939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.987990 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.987991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.988060 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.988065 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:07:22.988121 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:07:22.988169 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":80,"threads_started":1}
I20260926 01:07:22.988473 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.984155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.0.98.3:35427)
0926 01:07:22.984326 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:22.984331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:22.984528 (+ 197us) server_negotiation.cc:407] Connection header received
0926 01:07:22.984571 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:22.984573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:22.984587 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:22.984649 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:22.984833 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.985566 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.987067 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.987814 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.987893 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.987951 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:22.988348 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 01:07:22.988395 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260926 01:07:22.988428 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.983797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41135 (local address 127.0.0.1:47910)
0926 01:07:22.984181 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:22.984229 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:22.984317 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:22.984554 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:22.984558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:22.984560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:22.984647 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.984650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.986099 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.986100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.987267 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:22.987270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.988224 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.988224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:22.988290 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.988294 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:07:22.988327 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 01:07:22.988366 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":27,"thread_start_us":77,"threads_started":1}
I20260926 01:07:22.989267 14520 tablet_service.cc:1467] Processing CreateTablet for tablet 90b3457daa8a461e99c5d9e363e72ac5 (DEFAULT_TABLE table=test-table [id=e552fc31eaac448f8b77d6fb8317ccdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:22.989270 14252 tablet_service.cc:1467] Processing CreateTablet for tablet 90b3457daa8a461e99c5d9e363e72ac5 (DEFAULT_TABLE table=test-table [id=e552fc31eaac448f8b77d6fb8317ccdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:22.989671 14252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b3457daa8a461e99c5d9e363e72ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:22.989671 14520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b3457daa8a461e99c5d9e363e72ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:22.990585 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:22.983934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.0.98.2:41135)
0926 01:07:22.984224 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:22.984228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:22.984291 (+ 63us) server_negotiation.cc:407] Connection header received
0926 01:07:22.984419 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:22.984421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:22.984439 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:22.984723 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:22.984799 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.985972 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.987375 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:22.988127 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:22.988498 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:22.988713 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:22.988763 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 01:07:22.990513 (+ 1750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":88,"threads_started":1}
I20260926 01:07:22.991192 14386 tablet_service.cc:1467] Processing CreateTablet for tablet 90b3457daa8a461e99c5d9e363e72ac5 (DEFAULT_TABLE table=test-table [id=e552fc31eaac448f8b77d6fb8317ccdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:22.991542 14386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b3457daa8a461e99c5d9e363e72ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:22.993468 14603 tablet_bootstrap.cc:492] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Bootstrap starting.
I20260926 01:07:22.993710 14604 tablet_bootstrap.cc:492] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: Bootstrap starting.
I20260926 01:07:22.994465 14604 tablet_bootstrap.cc:654] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:22.995047 14603 tablet_bootstrap.cc:654] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:22.995025 14604 log.cc:824] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:22.995537 14605 tablet_bootstrap.cc:492] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: Bootstrap starting.
I20260926 01:07:22.995569 14603 log.cc:824] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:22.996083 14605 tablet_bootstrap.cc:654] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:22.996299 14603 tablet_bootstrap.cc:492] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: No bootstrap required, opened a new log
I20260926 01:07:22.996362 14603 ts_tablet_manager.cc:1397] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:07:22.996548 14605 log.cc:824] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:22.997255 14605 tablet_bootstrap.cc:492] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: No bootstrap required, opened a new log
I20260926 01:07:22.997267 14604 tablet_bootstrap.cc:492] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: No bootstrap required, opened a new log
I20260926 01:07:22.997329 14604 ts_tablet_manager.cc:1397] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 01:07:22.997329 14605 ts_tablet_manager.cc:1397] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:07:22.998023 14603 raft_consensus.cc:348] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:22.998174 14603 raft_consensus.cc:374] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:22.998205 14603 raft_consensus.cc:729] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0431139c48aa4ca88c3246a73f228cce, State: Initialized, Role: FOLLOWER
I20260926 01:07:22.998351 14603 consensus_queue.cc:260] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:22.998679 14318 heartbeater.cc:503] Master 127.0.98.62:45651 was elected leader, sending a full tablet report...
I20260926 01:07:22.998680 14603 ts_tablet_manager.cc:1428] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:07:22.999322 14604 raft_consensus.cc:348] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:22.999322 14605 raft_consensus.cc:348] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:22.999527 14604 raft_consensus.cc:374] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:22.999527 14605 raft_consensus.cc:374] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:22.999569 14604 raft_consensus.cc:729] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8353cc86dfca458294fac6c8c1ccba52, State: Initialized, Role: FOLLOWER
I20260926 01:07:22.999568 14605 raft_consensus.cc:729] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183c053291a0441b9b59751807b2d040, State: Initialized, Role: FOLLOWER
I20260926 01:07:22.999684 14604 consensus_queue.cc:260] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:22.999717 14605 consensus_queue.cc:260] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:22.999976 14605 ts_tablet_manager.cc:1428] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:07:22.999976 14604 ts_tablet_manager.cc:1428] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 01:07:23.000104 14586 heartbeater.cc:503] Master 127.0.98.62:45651 was elected leader, sending a full tablet report...
I20260926 01:07:23.000118 14452 heartbeater.cc:503] Master 127.0.98.62:45651 was elected leader, sending a full tablet report...
W20260926 01:07:23.038167 14453 tablet.cc:2369] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:07:23.121371 14319 tablet.cc:2369] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:07:23.137531 14611 raft_consensus.cc:482] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:07:23.137897 14611 raft_consensus.cc:504] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:23.138373 14611 leader_election.cc:288] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0431139c48aa4ca88c3246a73f228cce (127.0.98.1:38111), 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427)
I20260926 01:07:23.142736 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.138872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48425 (local address 127.0.98.3:35427)
0926 01:07:23.138934 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.138936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.139147 (+ 211us) server_negotiation.cc:407] Connection header received
0926 01:07:23.139248 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.139250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.139264 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.139341 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:23.139656 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.140320 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.141702 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.142286 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.142400 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.142414 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.142602 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.142664 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:07:23.142742 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.138786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35427 (local address 127.0.98.2:48425)
0926 01:07:23.138971 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.138980 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:23.139078 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:23.139381 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:23.139382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:23.139385 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:23.139457 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.139458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.140395 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.140396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.141565 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.141569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.142435 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.142436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.142514 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.142520 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 01:07:23.142586 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 01:07:23.142639 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":112,"threads_started":1}
I20260926 01:07:23.142843 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.138648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.98.2:48341)
0926 01:07:23.138708 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.138721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:23.138770 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:23.138946 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:23.138947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:23.138949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:23.139025 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.139027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.140229 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.140230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.141565 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.141569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.142523 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.142524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.142599 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.142602 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:07:23.142683 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 01:07:23.142722 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:07:23.143004 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.138717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48341 (local address 127.0.98.1:38111)
0926 01:07:23.138772 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.138774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.138824 (+ 50us) server_negotiation.cc:407] Connection header received
0926 01:07:23.138854 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.138855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.138865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.138923 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:23.139238 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.140129 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.141832 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.142451 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.142516 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.142525 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.142704 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.142750 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:07:23.143149 14540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b3457daa8a461e99c5d9e363e72ac5" candidate_uuid: "183c053291a0441b9b59751807b2d040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8353cc86dfca458294fac6c8c1ccba52" is_pre_election: true
I20260926 01:07:23.143255 14272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b3457daa8a461e99c5d9e363e72ac5" candidate_uuid: "183c053291a0441b9b59751807b2d040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431139c48aa4ca88c3246a73f228cce" is_pre_election: true
I20260926 01:07:23.143316 14540 raft_consensus.cc:2454] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183c053291a0441b9b59751807b2d040 in term 0.
I20260926 01:07:23.143407 14272 raft_consensus.cc:2454] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183c053291a0441b9b59751807b2d040 in term 0.
I20260926 01:07:23.143532 14341 leader_election.cc:302] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183c053291a0441b9b59751807b2d040, 8353cc86dfca458294fac6c8c1ccba52; no voters:
I20260926 01:07:23.143672 14611 raft_consensus.cc:2781] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:07:23.143734 14611 raft_consensus.cc:482] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:07:23.143757 14611 raft_consensus.cc:3037] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:23.144476 14611 raft_consensus.cc:504] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:23.144608 14611 leader_election.cc:288] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [CANDIDATE]: Term 1 election: Requested vote from peers 0431139c48aa4ca88c3246a73f228cce (127.0.98.1:38111), 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427)
I20260926 01:07:23.144819 14272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b3457daa8a461e99c5d9e363e72ac5" candidate_uuid: "183c053291a0441b9b59751807b2d040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431139c48aa4ca88c3246a73f228cce"
I20260926 01:07:23.144834 14540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b3457daa8a461e99c5d9e363e72ac5" candidate_uuid: "183c053291a0441b9b59751807b2d040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8353cc86dfca458294fac6c8c1ccba52"
I20260926 01:07:23.144901 14272 raft_consensus.cc:3037] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:23.144912 14540 raft_consensus.cc:3037] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:23.145637 14540 raft_consensus.cc:2454] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183c053291a0441b9b59751807b2d040 in term 1.
I20260926 01:07:23.145656 14272 raft_consensus.cc:2454] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183c053291a0441b9b59751807b2d040 in term 1.
I20260926 01:07:23.145812 14341 leader_election.cc:302] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0431139c48aa4ca88c3246a73f228cce, 183c053291a0441b9b59751807b2d040; no voters:
I20260926 01:07:23.145907 14611 raft_consensus.cc:2781] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:07:23.146059 14611 raft_consensus.cc:686] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 1 LEADER]: Becoming Leader. State: Replica: 183c053291a0441b9b59751807b2d040, State: Running, Role: LEADER
I20260926 01:07:23.146149 14611 consensus_queue.cc:237] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:23.147035 14127 catalog_manager.cc:5495] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 reported cstate change: term changed from 0 to 1, leader changed from <none> to 183c053291a0441b9b59751807b2d040 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "183c053291a0441b9b59751807b2d040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } health_report { overall_health: UNKNOWN } } }
I20260926 01:07:23.171455 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.168699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.0.98.2:41135)
0926 01:07:23.168759 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.168761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.168773 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:07:23.168828 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.168829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.168851 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.168910 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:07:23.169081 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.169597 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.170455 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.170655 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.170713 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.170985 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:07:23.171114 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:07:23.171198 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.171221 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.171248 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:07:23.173439 14540 raft_consensus.cc:1261] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52 [term 1 FOLLOWER]: Refusing update from remote peer 183c053291a0441b9b59751807b2d040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:07:23.173472 14272 raft_consensus.cc:1261] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Refusing update from remote peer 183c053291a0441b9b59751807b2d040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:07:23.173906 14613 consensus_queue.cc:1035] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:07:23.174094 14611 consensus_queue.cc:1035] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:07:23.176973 14618 mvcc.cc:205] Tried to move back new op lower bound from 7333416317635317760 to 7333416317527019520. Current Snapshot: MvccSnapshot[applied={T|T < 7333416317635317760}]
I20260926 01:07:23.177119 14617 mvcc.cc:205] Tried to move back new op lower bound from 7333416317635317760 to 7333416317527019520. Current Snapshot: MvccSnapshot[applied={T|T < 7333416317635317760}]
I20260926 01:07:23.179315 14126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:07:23.179420 14126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:07:23.180218 14126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Sending DeleteTablet for 3 replicas of tablet 90b3457daa8a461e99c5d9e363e72ac5
I20260926 01:07:23.180550 14386 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b3457daa8a461e99c5d9e363e72ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47910
I20260926 01:07:23.180703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 14189
I20260926 01:07:23.180724 14520 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b3457daa8a461e99c5d9e363e72ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51508
I20260926 01:07:23.180801 14252 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b3457daa8a461e99c5d9e363e72ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47126
I20260926 01:07:23.181248 14625 tablet_replica.cc:330] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: stopping tablet replica
I20260926 01:07:23.181360 14625 raft_consensus.cc:2229] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:07:23.181463 14625 raft_consensus.cc:2258] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:23.181916 14625 ts_tablet_manager.cc:1905] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:07:23.183760 14625 ts_tablet_manager.cc:1918] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:07:23.183848 14625 log.cc:1196] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/90b3457daa8a461e99c5d9e363e72ac5
I20260926 01:07:23.184126 14625 ts_tablet_manager.cc:1939] T 90b3457daa8a461e99c5d9e363e72ac5 P 183c053291a0441b9b59751807b2d040: Deleting consensus metadata
I20260926 01:07:23.184587 14114 catalog_manager.cc:4841] TS 183c053291a0441b9b59751807b2d040 (127.0.98.2:41135): tablet 90b3457daa8a461e99c5d9e363e72ac5 (table test-table [id=e552fc31eaac448f8b77d6fb8317ccdf]) successfully deleted
I20260926 01:07:23.186543 14626 tablet_replica.cc:330] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: stopping tablet replica
I20260926 01:07:23.186699 14626 raft_consensus.cc:2229] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:07:23.186832 14626 raft_consensus.cc:2258] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:23.187376 14626 ts_tablet_manager.cc:1905] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:07:23.187362 14113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38111 (error 108)
I20260926 01:07:23.187917 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.187774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.0.1:47136)
0926 01:07:23.187829 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.187864 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:07:23.187954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 14323
W20260926 01:07:23.188653 14113 catalog_manager.cc:4584] TS 0431139c48aa4ca88c3246a73f228cce (127.0.98.1:38111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90b3457daa8a461e99c5d9e363e72ac5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38111: connect: Connection refused (error 111)
I20260926 01:07:23.190539 14626 ts_tablet_manager.cc:1918] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:07:23.190639 14626 log.cc:1196] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/90b3457daa8a461e99c5d9e363e72ac5
I20260926 01:07:23.190904 14626 ts_tablet_manager.cc:1939] T 90b3457daa8a461e99c5d9e363e72ac5 P 8353cc86dfca458294fac6c8c1ccba52: Deleting consensus metadata
I20260926 01:07:23.191408 14113 catalog_manager.cc:4841] TS 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427): tablet 90b3457daa8a461e99c5d9e363e72ac5 (table test-table [id=e552fc31eaac448f8b77d6fb8317ccdf]) successfully deleted
I20260926 01:07:23.192610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 14457
I20260926 01:07:23.197476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 14095
I20260926 01:07:23.202325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45651
--webserver_interface=127.0.98.62
--webserver_port=38959
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:23.287354 14628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:23.287587 14628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:23.287612 14628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:23.288861 14628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:07:23.288918 14628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:23.288931 14628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:23.288942 14628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:07:23.288954 14628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:07:23.290614 14628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:23.290879 14628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:23.291122 14628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:23.300441 14634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:23.300460 14637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:23.300559 14635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:23.300704 14628 server_base.cc:1034] running on GCE node
I20260926 01:07:23.301108 14628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:23.301443 14628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:23.302593 14628 hybrid_clock.cc:648] HybridClock initialized: now 1790384843302579 us; error 35 us; skew 500 ppm
Sep 26 01:07:23 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:23.311647 14628 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:07:23.312206 14628 webserver.cc:460] Webserver started at http://127.0.98.62:38959/ using document root <none> and password file <none>
I20260926 01:07:23.312436 14628 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:23.312505 14628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:23.314185 14628 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:07:23.315052 14644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:23.315259 14628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:07:23.315392 14628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "4e76b9b16b835a7c01dc9959e374b1a4"
server_key_iv: "c6172454d04c7d50281fff6222895f83"
server_key_version: "encryptionkey@0"
I20260926 01:07:23.315847 14628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:23.332140 14628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:23.334834 14628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:23.334981 14628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:23.338831 14628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45651
I20260926 01:07:23.338866 14696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45651 every 8 connection(s)
I20260926 01:07:23.339253 14628 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:07:23.340796 14697 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:07:23.341486 14697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Bootstrap starting.
I20260926 01:07:23.347960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 14628
I20260926 01:07:23.348320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38111
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45651
--webserver_port=35109
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:07:23.350024 14697 log.cc:824] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:23.352217 14697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:07:23.352485 14697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Bootstrap complete.
I20260926 01:07:23.354732 14697 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:23.355005 14697 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf4c5aa7a3249999bbf8963ac7e3fff, State: Initialized, Role: FOLLOWER
I20260926 01:07:23.355165 14697 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:23.355232 14697 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:07:23.355288 14697 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:07:23.355333 14697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:07:23.356467 14697 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:23.356563 14697 leader_election.cc:302] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf4c5aa7a3249999bbf8963ac7e3fff; no voters:
I20260926 01:07:23.356726 14697 leader_election.cc:288] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:07:23.356778 14701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:07:23.356937 14701 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 2 LEADER]: Becoming Leader. State: Replica: cbf4c5aa7a3249999bbf8963ac7e3fff, State: Running, Role: LEADER
I20260926 01:07:23.357015 14697 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:07:23.357076 14701 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:23.357411 14701 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf4c5aa7a3249999bbf8963ac7e3fff. Latest consensus state: current_term: 2 leader_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } } }
I20260926 01:07:23.357501 14701 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:23.357390 14702 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } } }
I20260926 01:07:23.357949 14702 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:23.358098 14707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:07:23.358942 14707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:07:23.359099 14707 catalog_manager.cc:1259] Loaded cluster ID: 888096b31e384ae18d8a3baa697b3d41
I20260926 01:07:23.359302 14707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:07:23.359532 14707 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 01:07:23.362253 14707 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 01:07:23.367414 14707 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:07:23.367637 14707 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Loaded TSK: 0
I20260926 01:07:23.367846 14707 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:07:23.446139 14699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:23.446436 14699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:23.446487 14699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:23.447937 14699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:23.448001 14699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:23.448053 14699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:07:23.449757 14699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35109
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:23.450031 14699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:23.450258 14699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:23.455430 14727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:23.455523 14725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:23.455435 14724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:23.455881 14699 server_base.cc:1034] running on GCE node
I20260926 01:07:23.456081 14699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:23.456312 14699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:23.457463 14699 hybrid_clock.cc:648] HybridClock initialized: now 1790384843457437 us; error 36 us; skew 500 ppm
Sep 26 01:07:23 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:23.460276 14699 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:07:23.460764 14699 webserver.cc:460] Webserver started at http://127.0.98.1:35109/ using document root <none> and password file <none>
I20260926 01:07:23.460999 14699 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:23.461062 14699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:23.462602 14699 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:23.463359 14734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:23.463548 14699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:23.463647 14699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "0431139c48aa4ca88c3246a73f228cce"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "d89aa8acd101fe6ee5623e9bc7acb4f9"
server_key_iv: "2417e693987c51bf43ff6bd9a55750af"
server_key_version: "encryptionkey@0"
I20260926 01:07:23.464118 14699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:23.479339 14699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:23.481616 14699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:23.481776 14699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:23.481992 14699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:23.482462 14741 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:07:23.484340 14699 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:07:23.484391 14699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:07:23.484426 14699 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:07:23.485092 14699 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:07:23.485128 14699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:07:23.485216 14741 tablet_bootstrap.cc:492] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Bootstrap starting.
I20260926 01:07:23.490334 14699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38111
I20260926 01:07:23.490820 14699 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:07:23.493187 14848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38111 every 8 connection(s)
I20260926 01:07:23.494292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 14699
I20260926 01:07:23.494630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41135
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45651
--webserver_port=32829
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:07:23.495931 14741 log.cc:824] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:23.499071 14741 tablet_bootstrap.cc:492] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:07:23.499804 14741 tablet_bootstrap.cc:492] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Bootstrap complete.
Sep 26 01:07:23 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:23.501682 14741 ts_tablet_manager.cc:1397] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20260926 01:07:23.503748 14741 raft_consensus.cc:348] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:23.504187 14741 raft_consensus.cc:729] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0431139c48aa4ca88c3246a73f228cce, State: Initialized, Role: FOLLOWER
I20260926 01:07:23.504398 14741 consensus_queue.cc:260] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } }
I20260926 01:07:23.504853 14741 ts_tablet_manager.cc:1428] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 01:07:23.506271 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.491823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:40587)
0926 01:07:23.493343 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.493390 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:23.493927 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:23.494559 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:23.494563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:23.494908 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:23.495207 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.495216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.496955 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.496957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.498062 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.498066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.498968 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.498970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.499120 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.499747 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:23.499758 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:23.501697 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:23.504122 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:23.504131 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:23.504133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:23.504475 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:23.504809 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:23.504810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:23.504810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:23.504948 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:23.505510 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:23.505518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:23.505700 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 01:07:23.505850 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 01:07:23.506063 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1472,"thread_start_us":82,"threads_started":1}
I20260926 01:07:23.506317 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.491987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40587 (local address 127.0.98.62:45651)
0926 01:07:23.492146 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.492156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.493542 (+ 1386us) server_negotiation.cc:407] Connection header received
0926 01:07:23.494163 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.494171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.494234 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.494371 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:23.495354 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.496777 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.498173 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.498741 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.498846 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.501857 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:23.501863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:23.501863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:23.501879 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:23.503997 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:23.504584 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:23.504588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:23.504589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:23.504731 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:23.505027 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:23.505028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:23.505029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:23.505395 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:23.505492 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.505946 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.506117 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260926 01:07:23.508152 14849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45651
I20260926 01:07:23.508270 14849 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:23.508462 14849 heartbeater.cc:511] Master 127.0.98.62:45651 requested a full tablet report, sending...
I20260926 01:07:23.509255 14661 ts_manager.cc:194] Registered new tserver with Master: 0431139c48aa4ca88c3246a73f228cce (127.0.98.1:38111)
I20260926 01:07:23.510432 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40587
I20260926 01:07:23.511461 14849 heartbeater.cc:503] Master 127.0.98.62:45651 was elected leader, sending a full tablet report...
Sep 26 01:07:23 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:07:23.519505 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.509926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.0.1:47140)
0926 01:07:23.510405 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.510448 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:23.510528 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:23.510754 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:23.510759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:23.511114 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:23.511179 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.511183 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.512655 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.512656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.513367 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.513371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.514042 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.514047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.514121 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.514679 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:23.514683 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:23.515975 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:23.518127 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:23.518139 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:23.518142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:23.518373 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:23.518709 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:23.518710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:23.518711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:23.518781 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:23.519143 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:23.519151 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:23.519216 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 01:07:23.519294 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 01:07:23.519415 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":113,"threads_started":1}
I20260926 01:07:23.519680 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.510165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.0.98.1:38111)
0926 01:07:23.510349 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.510354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.510607 (+ 253us) server_negotiation.cc:407] Connection header received
0926 01:07:23.510654 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.510656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.510673 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.510751 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:23.511271 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.512560 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.513463 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.513940 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.514017 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.516073 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:23.516085 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:23.516085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:23.516094 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:23.517971 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:23.518520 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:23.518524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:23.518525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:23.518596 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:23.518853 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:23.518853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:23.518854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:23.519066 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:23.519149 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.519363 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.519473 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260926 01:07:23.520063 14764 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b3457daa8a461e99c5d9e363e72ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47140
I20260926 01:07:23.520637 14858 tablet_replica.cc:330] stopping tablet replica
I20260926 01:07:23.520750 14858 raft_consensus.cc:2229] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:07:23.520817 14858 raft_consensus.cc:2258] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:23.521154 14858 ts_tablet_manager.cc:1905] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:07:23.523829 14858 ts_tablet_manager.cc:1918] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:07:23.523942 14858 log.cc:1196] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/90b3457daa8a461e99c5d9e363e72ac5
I20260926 01:07:23.524166 14858 ts_tablet_manager.cc:1939] T 90b3457daa8a461e99c5d9e363e72ac5 P 0431139c48aa4ca88c3246a73f228cce: Deleting consensus metadata
I20260926 01:07:23.525487 14647 catalog_manager.cc:4841] TS 0431139c48aa4ca88c3246a73f228cce (127.0.98.1:38111): tablet 90b3457daa8a461e99c5d9e363e72ac5 (table test-table [id=e552fc31eaac448f8b77d6fb8317ccdf]) successfully deleted
W20260926 01:07:23.588699 14853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:23.589085 14853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:23.589119 14853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:23.590783 14853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:23.590860 14853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:23.590909 14853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:07:23.592758 14853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32829
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:23.592998 14853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:23.593225 14853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:23.598367 14863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:23.598361 14864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:23.598353 14866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:23.598692 14853 server_base.cc:1034] running on GCE node
I20260926 01:07:23.598886 14853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:23.599130 14853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:23.600294 14853 hybrid_clock.cc:648] HybridClock initialized: now 1790384843600278 us; error 35 us; skew 500 ppm
Sep 26 01:07:23 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:23.602945 14853 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:07:23.603392 14853 webserver.cc:460] Webserver started at http://127.0.98.2:32829/ using document root <none> and password file <none>
I20260926 01:07:23.603623 14853 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:23.603683 14853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:23.605170 14853 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:23.605814 14873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:23.605969 14853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:07:23.606048 14853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "183c053291a0441b9b59751807b2d040"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "1a5ab8d2e1285482f88fef5a07bcf396"
server_key_iv: "0123e5e343770060f907214c38f49efb"
server_key_version: "encryptionkey@0"
I20260926 01:07:23.606428 14853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:23.626924 14853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:23.629104 14853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:23.629225 14853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:23.629459 14853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:23.629760 14853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:23.629792 14853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:23.629814 14853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:23.629824 14853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:23.635669 14853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41135
I20260926 01:07:23.635767 14986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41135 every 8 connection(s)
I20260926 01:07:23.636075 14853 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:07:23.642184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 14853
Sep 26 01:07:23 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:23.642509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35427
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45651
--webserver_port=39855
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:07:23.645805 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.636781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37483 (local address 127.0.98.62:45651)
0926 01:07:23.636829 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.636831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.637055 (+ 224us) server_negotiation.cc:407] Connection header received
0926 01:07:23.637552 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.637554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.637571 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.637629 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:23.638655 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.639152 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.640176 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.640378 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.640437 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.642863 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:23.642865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:23.642865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:23.642874 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:23.644315 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:23.644752 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:23.644754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:23.644754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:23.644813 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:23.645134 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:23.645135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:23.645136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:23.645282 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:23.645352 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.645604 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.645706 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:07:23.645938 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.636779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:37483)
0926 01:07:23.636924 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.636961 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:23.637453 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:23.637703 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:23.637707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:23.638295 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:23.638553 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.638559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.639246 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.639247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.640085 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.640087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.640458 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.640459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.640598 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.641109 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:23.641118 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:23.642730 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:23.644406 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:23.644416 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:23.644419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:23.644678 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:23.644930 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:23.644930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:23.644931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:23.645059 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:23.645344 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:23.645347 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:23.645488 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 01:07:23.645574 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 01:07:23.645719 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260926 01:07:23.646502 14987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45651
I20260926 01:07:23.646602 14987 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:23.646786 14987 heartbeater.cc:511] Master 127.0.98.62:45651 requested a full tablet report, sending...
I20260926 01:07:23.647223 14661 ts_manager.cc:194] Registered new tserver with Master: 183c053291a0441b9b59751807b2d040 (127.0.98.2:41135)
I20260926 01:07:23.647740 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37483
W20260926 01:07:23.727592 14990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:23.727825 14990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:23.727854 14990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:23.729169 14990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:23.729228 14990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:23.729259 14990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:07:23.730939 14990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39855
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:23.731184 14990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:23.731389 14990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:23.737124 14996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:23.736999 14995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:23.736948 14998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:23.737021 14990 server_base.cc:1034] running on GCE node
I20260926 01:07:23.737498 14990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:23.737789 14990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:23.738983 14990 hybrid_clock.cc:648] HybridClock initialized: now 1790384843738963 us; error 30 us; skew 500 ppm
Sep 26 01:07:23 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:23.741494 14990 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:07:23.741988 14990 webserver.cc:460] Webserver started at http://127.0.98.3:39855/ using document root <none> and password file <none>
I20260926 01:07:23.742226 14990 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:23.742285 14990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:23.743752 14990 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:23.744446 15005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:23.744704 14990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:23.744807 14990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "8353cc86dfca458294fac6c8c1ccba52"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "bd91811126faa0c48206a7f097234c76"
server_key_iv: "a03d3b312e721bf15e482ee454bc6f50"
server_key_version: "encryptionkey@0"
I20260926 01:07:23.745203 14990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:23.760769 14990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:23.763098 14990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:23.763218 14990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:23.763456 14990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:23.763746 14990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:23.763778 14990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:23.763800 14990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:23.763813 14990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:23.769528 14990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35427
I20260926 01:07:23.769593 15118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35427 every 8 connection(s)
I20260926 01:07:23.769941 14990 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:07:23 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:23.778290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 14990
I20260926 01:07:23.780181 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.770692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41945 (local address 127.0.98.62:45651)
0926 01:07:23.770745 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.770747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.770954 (+ 207us) server_negotiation.cc:407] Connection header received
0926 01:07:23.771518 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.771520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.771533 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.771587 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:23.772552 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.773043 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.774320 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.774600 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.774667 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.777127 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:23.777129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:23.777129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:23.777139 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:23.778806 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:23.779245 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:23.779246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:23.779246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:23.779305 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:23.779536 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:23.779537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:23.779537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:23.779663 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:23.779722 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.779996 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.780085 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:07:23.780392 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.770665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:41945)
0926 01:07:23.770837 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.770873 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:23.771371 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:23.771672 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:23.771676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:23.772252 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:23.772450 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.772455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.773139 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.773140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.774208 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.774211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.774684 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.774686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.774831 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.775366 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:23.775375 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:23.777011 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:23.778894 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:23.778905 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:23.778908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:23.779173 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:23.779363 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:23.779364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:23.779364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:23.779479 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:23.779720 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:23.779727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:23.779876 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 01:07:23.779967 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 01:07:23.780172 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260926 01:07:23.780898 15119 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45651
I20260926 01:07:23.781035 15119 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:23.781245 15119 heartbeater.cc:511] Master 127.0.98.62:45651 requested a full tablet report, sending...
I20260926 01:07:23.781749 14661 ts_manager.cc:194] Registered new tserver with Master: 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427)
I20260926 01:07:23.782112 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41945
I20260926 01:07:23.785164 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.778785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.0.98.62:45651)
0926 01:07:23.778914 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.778916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.778924 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:07:23.779016 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.779018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.779028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.779082 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:23.779636 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.780139 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.781232 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.781509 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.781575 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.782887 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:23.782889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:23.782889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:23.782897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:23.784081 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:23.784398 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:23.784399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:23.784399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:23.784453 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:23.784616 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:23.784617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:23.784617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:23.784732 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:23.784901 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.784970 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.785060 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260926 01:07:23.786082 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:07:23.788774 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.786471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.0.98.62:45651)
0926 01:07:23.786525 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.786527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.786658 (+ 131us) server_negotiation.cc:407] Connection header received
0926 01:07:23.786716 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.786716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.786748 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.786805 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:07:23.786885 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.787387 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.788128 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.788340 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.788394 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.788494 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:07:23.788595 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:07:23.788650 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.788678 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.788699 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:07:23.789103 14661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:07:23.790755 14661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:07:23.793056 14764 tablet_service.cc:1467] Processing CreateTablet for tablet e6e9acd2681147d6bfbbc65cf7aa544d (DEFAULT_TABLE table=test-table [id=a31bf1cdf31b4b8184ea716454841c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:23.793283 14764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e9acd2681147d6bfbbc65cf7aa544d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:23.794782 14741 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Bootstrap starting.
I20260926 01:07:23.795148 14741 tablet_bootstrap.cc:654] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:23.796482 14741 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: No bootstrap required, opened a new log
I20260926 01:07:23.796546 14741 ts_tablet_manager.cc:1397] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:07:23.796718 14741 raft_consensus.cc:348] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:23.796785 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.792618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41135 (local address 127.0.0.1:47928)
0926 01:07:23.792674 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.792686 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:23.792726 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:23.793085 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:23.793087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:23.793089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:23.793143 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.793145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.794616 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.794618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.795597 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.795601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.796470 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.796471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.796552 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.796592 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 01:07:23.796655 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 01:07:23.796709 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 01:07:23.796795 14741 raft_consensus.cc:374] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:23.796854 14741 raft_consensus.cc:729] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0431139c48aa4ca88c3246a73f228cce, State: Initialized, Role: FOLLOWER
I20260926 01:07:23.796813 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.792721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.0.98.2:41135)
0926 01:07:23.792911 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.792916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.792926 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:07:23.792960 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.792962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.792975 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.793062 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:23.793237 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.794494 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.795686 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.796376 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.796457 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.796531 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.796682 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.796732 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260926 01:07:23.796912 14741 consensus_queue.cc:260] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:23.797014 14741 ts_tablet_manager.cc:1428] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:23.797063 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.792621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35427 (local address 127.0.0.1:51510)
0926 01:07:23.792807 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.792814 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:23.792849 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:23.793438 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:23.793439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:23.793442 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:23.793481 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.793483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.794915 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.794916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.796001 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.796003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.796860 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.796861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.796921 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.796925 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:07:23.796956 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 01:07:23.796996 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":19,"thread_start_us":115,"threads_started":1}
I20260926 01:07:23.797379 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.792721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.0.98.3:35427)
0926 01:07:23.793221 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.793225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.793234 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:07:23.793269 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.793272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.793287 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.793357 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:23.793561 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.794806 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.796086 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.796764 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.797160 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.797237 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.797261 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.797304 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":88,"threads_started":1}
I20260926 01:07:23.797996 15053 tablet_service.cc:1467] Processing CreateTablet for tablet e6e9acd2681147d6bfbbc65cf7aa544d (DEFAULT_TABLE table=test-table [id=a31bf1cdf31b4b8184ea716454841c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:23.798170 14921 tablet_service.cc:1467] Processing CreateTablet for tablet e6e9acd2681147d6bfbbc65cf7aa544d (DEFAULT_TABLE table=test-table [id=a31bf1cdf31b4b8184ea716454841c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:23.798319 15053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e9acd2681147d6bfbbc65cf7aa544d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:23.798563 14921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e9acd2681147d6bfbbc65cf7aa544d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:23.801102 15130 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Bootstrap starting.
I20260926 01:07:23.801100 15129 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Bootstrap starting.
I20260926 01:07:23.801751 15130 tablet_bootstrap.cc:654] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:23.801787 15129 tablet_bootstrap.cc:654] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:23.802208 15130 log.cc:824] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:23.802234 15129 log.cc:824] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:23.802950 15130 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: No bootstrap required, opened a new log
I20260926 01:07:23.802969 15129 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: No bootstrap required, opened a new log
I20260926 01:07:23.803016 15129 ts_tablet_manager.cc:1397] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:07:23.803021 15130 ts_tablet_manager.cc:1397] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:07:23.804538 15130 raft_consensus.cc:348] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:23.804538 15129 raft_consensus.cc:348] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:23.804733 15129 raft_consensus.cc:374] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:23.804733 15130 raft_consensus.cc:374] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:23.804762 15129 raft_consensus.cc:729] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8353cc86dfca458294fac6c8c1ccba52, State: Initialized, Role: FOLLOWER
I20260926 01:07:23.804781 15130 raft_consensus.cc:729] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183c053291a0441b9b59751807b2d040, State: Initialized, Role: FOLLOWER
I20260926 01:07:23.804893 15130 consensus_queue.cc:260] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:23.804890 15129 consensus_queue.cc:260] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:23.805171 15130 ts_tablet_manager.cc:1428] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:07:23.805186 15129 ts_tablet_manager.cc:1428] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:07:23.805226 14987 heartbeater.cc:503] Master 127.0.98.62:45651 was elected leader, sending a full tablet report...
I20260926 01:07:23.805308 15119 heartbeater.cc:503] Master 127.0.98.62:45651 was elected leader, sending a full tablet report...
I20260926 01:07:23.876605 15133 raft_consensus.cc:482] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:07:23.876756 15133 raft_consensus.cc:504] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:23.877122 15133 leader_election.cc:288] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427), 0431139c48aa4ca88c3246a73f228cce (127.0.98.1:38111)
I20260926 01:07:23.880335 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.877413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56967 (local address 127.0.98.3:35427)
0926 01:07:23.877467 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.877469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.877482 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:07:23.877569 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.877570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.877583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.877625 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:23.877799 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.878356 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.879461 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.879934 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.879988 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.879997 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.880239 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.880277 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:07:23.880352 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.877352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35427 (local address 127.0.98.2:56967)
0926 01:07:23.877416 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.877427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:23.877500 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:23.877641 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:23.877642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:23.877644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:23.877723 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.877725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.878524 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.878525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.879362 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.879365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.880062 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.880063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.880153 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.880158 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:07:23.880220 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 01:07:23.880267 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 01:07:23.880676 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.877668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58491 (local address 127.0.98.1:38111)
0926 01:07:23.877720 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.877722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.877731 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:07:23.877762 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.877763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.877789 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.877868 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:23.878051 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.878705 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.879762 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.880291 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.880355 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.880445 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.880555 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.880604 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:07:23.880690 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.877469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.98.2:58491)
0926 01:07:23.877655 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.877667 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:23.877724 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:23.877877 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:23.877877 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:23.877879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:23.877943 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.877944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.878800 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.878801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.879656 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:23.879659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.880368 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.880369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:23.880439 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.880443 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:07:23.880528 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 01:07:23.880572 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260926 01:07:23.880776 15073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e9acd2681147d6bfbbc65cf7aa544d" candidate_uuid: "183c053291a0441b9b59751807b2d040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8353cc86dfca458294fac6c8c1ccba52" is_pre_election: true
I20260926 01:07:23.880961 15073 raft_consensus.cc:2454] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183c053291a0441b9b59751807b2d040 in term 0.
I20260926 01:07:23.881263 14876 leader_election.cc:302] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183c053291a0441b9b59751807b2d040, 8353cc86dfca458294fac6c8c1ccba52; no voters:
I20260926 01:07:23.881417 15133 raft_consensus.cc:2781] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:07:23.881318 14803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e9acd2681147d6bfbbc65cf7aa544d" candidate_uuid: "183c053291a0441b9b59751807b2d040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431139c48aa4ca88c3246a73f228cce" is_pre_election: true
I20260926 01:07:23.881461 15133 raft_consensus.cc:482] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:07:23.881484 15133 raft_consensus.cc:3037] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:23.881532 14803 raft_consensus.cc:2454] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183c053291a0441b9b59751807b2d040 in term 0.
I20260926 01:07:23.882263 15133 raft_consensus.cc:504] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:23.882411 15133 leader_election.cc:288] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [CANDIDATE]: Term 1 election: Requested vote from peers 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427), 0431139c48aa4ca88c3246a73f228cce (127.0.98.1:38111)
I20260926 01:07:23.882740 15073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e9acd2681147d6bfbbc65cf7aa544d" candidate_uuid: "183c053291a0441b9b59751807b2d040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8353cc86dfca458294fac6c8c1ccba52"
I20260926 01:07:23.882784 14803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e9acd2681147d6bfbbc65cf7aa544d" candidate_uuid: "183c053291a0441b9b59751807b2d040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431139c48aa4ca88c3246a73f228cce"
I20260926 01:07:23.882858 14803 raft_consensus.cc:3037] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:23.882839 15073 raft_consensus.cc:3037] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:23.883566 14803 raft_consensus.cc:2454] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183c053291a0441b9b59751807b2d040 in term 1.
I20260926 01:07:23.883621 15073 raft_consensus.cc:2454] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183c053291a0441b9b59751807b2d040 in term 1.
I20260926 01:07:23.883745 14876 leader_election.cc:302] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0431139c48aa4ca88c3246a73f228cce, 183c053291a0441b9b59751807b2d040; no voters:
I20260926 01:07:23.883850 15133 raft_consensus.cc:2781] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:07:23.884001 15133 raft_consensus.cc:686] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 1 LEADER]: Becoming Leader. State: Replica: 183c053291a0441b9b59751807b2d040, State: Running, Role: LEADER
I20260926 01:07:23.884099 15133 consensus_queue.cc:237] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:23.885097 14660 catalog_manager.cc:5495] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 reported cstate change: term changed from 0 to 1, leader changed from <none> to 183c053291a0441b9b59751807b2d040 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "183c053291a0441b9b59751807b2d040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } health_report { overall_health: UNKNOWN } } }
W20260926 01:07:23.886600 14988 tablet.cc:2369] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:07:23.894974 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.892236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.0.98.2:41135)
0926 01:07:23.892332 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:23.892334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:23.892343 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:07:23.892384 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:23.892385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:23.892407 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:23.892475 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:07:23.892736 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.893248 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.894023 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:23.894237 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:23.894301 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:23.894473 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:07:23.894588 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:07:23.894665 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:23.894850 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 01:07:23.894882 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 01:07:23.897008 14803 raft_consensus.cc:1261] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Refusing update from remote peer 183c053291a0441b9b59751807b2d040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:07:23.897008 15073 raft_consensus.cc:1261] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 1 FOLLOWER]: Refusing update from remote peer 183c053291a0441b9b59751807b2d040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:07:23.897464 15136 consensus_queue.cc:1035] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:07:23.897598 15133 consensus_queue.cc:1035] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:07:23.899914 15138 mvcc.cc:205] Tried to move back new op lower bound from 7333416320598966272 to 7333416320549945344. Current Snapshot: MvccSnapshot[applied={T|T < 7333416320598966272}]
I20260926 01:07:23.900130 15140 mvcc.cc:205] Tried to move back new op lower bound from 7333416320598966272 to 7333416320549945344. Current Snapshot: MvccSnapshot[applied={T|T < 7333416320598966272}]
I20260926 01:07:23.900147 15139 mvcc.cc:205] Tried to move back new op lower bound from 7333416320598966272 to 7333416320549945344. Current Snapshot: MvccSnapshot[applied={T|T < 7333416320598966272}]
I20260926 01:07:23.902350 14660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:07:23.902482 14660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:07:23.903205 14660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Sending DeleteTablet for 3 replicas of tablet e6e9acd2681147d6bfbbc65cf7aa544d
I20260926 01:07:23.903496 14921 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e9acd2681147d6bfbbc65cf7aa544d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47928
I20260926 01:07:23.903610 14764 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e9acd2681147d6bfbbc65cf7aa544d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47140
I20260926 01:07:23.903602 15053 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e9acd2681147d6bfbbc65cf7aa544d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51510
I20260926 01:07:23.903700 14858 tablet_replica.cc:330] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: stopping tablet replica
I20260926 01:07:23.903740 14858 raft_consensus.cc:2229] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:07:23.903761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 14699
I20260926 01:07:23.903784 14858 raft_consensus.cc:2258] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:23.904368 15149 tablet_replica.cc:330] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: stopping tablet replica
I20260926 01:07:23.904484 15149 raft_consensus.cc:2229] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:07:23.904592 15149 raft_consensus.cc:2258] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:23.905053 15149 ts_tablet_manager.cc:1905] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:07:23.906737 15149 ts_tablet_manager.cc:1918] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:07:23.906818 15149 log.cc:1196] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/e6e9acd2681147d6bfbbc65cf7aa544d
I20260926 01:07:23.907195 15149 ts_tablet_manager.cc:1939] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Deleting consensus metadata
I20260926 01:07:23.907708 14647 catalog_manager.cc:4841] TS 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427): tablet e6e9acd2681147d6bfbbc65cf7aa544d (table test-table [id=a31bf1cdf31b4b8184ea716454841c11]) successfully deleted
W20260926 01:07:23.910249 14647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38111 (error 108)
I20260926 01:07:23.910487 15148 tablet_replica.cc:330] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: stopping tablet replica
I20260926 01:07:23.910625 15148 raft_consensus.cc:2229] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:07:23.910746 15148 raft_consensus.cc:2258] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:23.910929 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.910807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.0.1:47156)
0926 01:07:23.910859 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.910892 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:07:23.910972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 14853
W20260926 01:07:23.911054 14647 catalog_manager.cc:4584] TS 0431139c48aa4ca88c3246a73f228cce (127.0.98.1:38111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6e9acd2681147d6bfbbc65cf7aa544d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38111: connect: Connection refused (error 111)
I20260926 01:07:23.915954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 14990
I20260926 01:07:23.916067 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:23.915934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41135 (local address 127.0.0.1:47944)
0926 01:07:23.915990 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:23.916023 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:07:23.919839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 14628
I20260926 01:07:23.925203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45651
--webserver_interface=127.0.98.62
--webserver_port=38959
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:24.007054 15151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:24.007288 15151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:24.007313 15151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:24.008572 15151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:07:24.008627 15151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:24.008641 15151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:24.008651 15151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:07:24.008663 15151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:07:24.010207 15151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:24.010475 15151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:24.010711 15151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:24.015673 15157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:24.015704 15156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:24.015683 15159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:24.015872 15151 server_base.cc:1034] running on GCE node
I20260926 01:07:24.016185 15151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:24.016476 15151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:24.017663 15151 hybrid_clock.cc:648] HybridClock initialized: now 1790384844017646 us; error 29 us; skew 500 ppm
Sep 26 01:07:24 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:24.020203 15151 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:07:24.020689 15151 webserver.cc:460] Webserver started at http://127.0.98.62:38959/ using document root <none> and password file <none>
I20260926 01:07:24.020897 15151 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:24.020946 15151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:24.022320 15151 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:24.023151 15166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:24.023435 15151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:24.023558 15151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "4e76b9b16b835a7c01dc9959e374b1a4"
server_key_iv: "c6172454d04c7d50281fff6222895f83"
server_key_version: "encryptionkey@0"
I20260926 01:07:24.023945 15151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:24.034937 15151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:24.037230 15151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:24.037379 15151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:24.041420 15151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45651
I20260926 01:07:24.041500 15218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45651 every 8 connection(s)
I20260926 01:07:24.041855 15151 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:07:24.043545 15219 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:07:24.044229 15219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Bootstrap starting.
I20260926 01:07:24.050195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 15151
I20260926 01:07:24.050529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38111
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45651
--webserver_port=35109
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:07:24.052516 15219 log.cc:824] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:24.054788 15219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:07:24.055092 15219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff: Bootstrap complete.
I20260926 01:07:24.057173 15219 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:24.057408 15219 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf4c5aa7a3249999bbf8963ac7e3fff, State: Initialized, Role: FOLLOWER
I20260926 01:07:24.057550 15219 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:24.057621 15219 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:07:24.057663 15219 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:07:24.057713 15219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:07:24.058804 15219 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:24.058903 15219 leader_election.cc:302] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf4c5aa7a3249999bbf8963ac7e3fff; no voters:
I20260926 01:07:24.059078 15219 leader_election.cc:288] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 01:07:24.059126 15223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 3 FOLLOWER]: Leader election won for term 3
I20260926 01:07:24.059239 15223 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [term 3 LEADER]: Becoming Leader. State: Replica: cbf4c5aa7a3249999bbf8963ac7e3fff, State: Running, Role: LEADER
I20260926 01:07:24.059325 15223 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260926 01:07:24.059358 15219 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:07:24.059616 15224 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } } }
I20260926 01:07:24.059619 15223 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf4c5aa7a3249999bbf8963ac7e3fff. Latest consensus state: current_term: 3 leader_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf4c5aa7a3249999bbf8963ac7e3fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } } }
I20260926 01:07:24.059834 15223 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:24.059813 15224 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf4c5aa7a3249999bbf8963ac7e3fff [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:24.060768 15231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:07:24.061590 15231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:07:24.061697 15231 catalog_manager.cc:1259] Loaded cluster ID: 888096b31e384ae18d8a3baa697b3d41
I20260926 01:07:24.061726 15231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 01:07:24.061929 15231 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff201b4e9ad google::LogMessage::Fail() at ??:0
@ 0x7ff201b5188a google::LogMessage::SendToLog() at ??:0
@ 0x7ff201b4e4de google::LogMessage::Flush() at ??:0
@ 0x7ff201b4fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff209a6f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff209a788b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff203a1c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff203a11954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff2048756db start_thread at ??:0
@ 0x7ff20064c61f clone at ??:0
W20260926 01:07:24.146603 15221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:24.146821 15221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:24.146852 15221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:24.148505 15221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:24.148571 15221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:24.148599 15221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:07:24.150290 15221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35109
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:24.150599 15221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:24.150833 15221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:24.155936 15256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:24.155989 15255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:24.156145 15221 server_base.cc:1034] running on GCE node
W20260926 01:07:24.155943 15258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:24.156388 15221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:24.156719 15221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:24.157914 15221 hybrid_clock.cc:648] HybridClock initialized: now 1790384844157891 us; error 36 us; skew 500 ppm
Sep 26 01:07:24 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:24.161331 15221 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:07:24.161966 15221 webserver.cc:460] Webserver started at http://127.0.98.1:35109/ using document root <none> and password file <none>
I20260926 01:07:24.162245 15221 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:24.162320 15221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:24.164001 15221 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:24.164844 15265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:24.165040 15221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:24.165124 15221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "0431139c48aa4ca88c3246a73f228cce"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "d89aa8acd101fe6ee5623e9bc7acb4f9"
server_key_iv: "2417e693987c51bf43ff6bd9a55750af"
server_key_version: "encryptionkey@0"
I20260926 01:07:24.165555 15221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:24.177556 15221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:24.180619 15221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:24.180785 15221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:24.181056 15221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:24.181524 15272 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:07:24.183024 15221 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:07:24.183092 15221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:07:24.183128 15221 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:07:24.184058 15221 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:07:24.184108 15221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:07:24.184190 15272 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Bootstrap starting.
I20260926 01:07:24.191435 15221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38111
I20260926 01:07:24.191547 15379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38111 every 8 connection(s)
I20260926 01:07:24.191998 15221 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:07:24.194619 15272 log.cc:824] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:24.195725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 15221
I20260926 01:07:24.196023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41135
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45651
--webserver_port=32829
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:07:24.197257 15272 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:07:24.197566 15272 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Bootstrap complete.
I20260926 01:07:24.198316 15272 ts_tablet_manager.cc:1397] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260926 01:07:24.199743 15272 raft_consensus.cc:348] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:24.200091 15272 raft_consensus.cc:729] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0431139c48aa4ca88c3246a73f228cce, State: Initialized, Role: FOLLOWER
I20260926 01:07:24.200278 15272 consensus_queue.cc:260] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:24.200682 15272 ts_tablet_manager.cc:1428] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
W20260926 01:07:24.279872 15384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:24.280125 15384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:24.280174 15384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:24.280468 15382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:07:24.192908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:47113)
0926 01:07:24.193142 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.193199 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:24.194112 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:24.280268 (+ 86156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
W20260926 01:07:24.280701 15380 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45651 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45651: Client connection negotiation failed: client connection to 127.0.98.62:45651: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:07:24.281088 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:24.280931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:41537)
0926 01:07:24.281005 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.281062 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 01:07:24.281490 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:24.281395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:47213)
0926 01:07:24.281439 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.281465 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:07:24.281603 15380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:07:24.281981 15384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:24.282030 15384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:24.282084 15384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:07:24.283866 15384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32829
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:24.284153 15384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:24.284381 15384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:24.289518 15393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:24.289551 15384 server_base.cc:1034] running on GCE node
W20260926 01:07:24.289512 15391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:24.289517 15390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:24.289943 15384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:24.290225 15384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:24.291385 15384 hybrid_clock.cc:648] HybridClock initialized: now 1790384844291337 us; error 58 us; skew 500 ppm
Sep 26 01:07:24 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:24.294498 15384 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:07:24.295020 15384 webserver.cc:460] Webserver started at http://127.0.98.2:32829/ using document root <none> and password file <none>
I20260926 01:07:24.295276 15384 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:24.295327 15384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:24.296866 15384 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:24.297904 15400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:24.298267 15384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:24.298408 15384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "183c053291a0441b9b59751807b2d040"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "1a5ab8d2e1285482f88fef5a07bcf396"
server_key_iv: "0123e5e343770060f907214c38f49efb"
server_key_version: "encryptionkey@0"
I20260926 01:07:24.298804 15384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:24.326474 15384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:24.328766 15384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:24.328912 15384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:24.329140 15384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:24.329561 15407 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:07:24.331133 15384 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:07:24.331203 15384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:07:24.331243 15384 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:07:24.331943 15384 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:07:24.331977 15384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:07:24.332048 15407 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Bootstrap starting.
I20260926 01:07:24.339435 15384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41135
I20260926 01:07:24.339533 15514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41135 every 8 connection(s)
I20260926 01:07:24.339890 15384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:07:24.341048 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:24.340532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:54731)
0926 01:07:24.340717 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.340824 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
W20260926 01:07:24.341353 15515 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45651 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45651: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
I20260926 01:07:24.341729 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:24.341604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:36157)
0926 01:07:24.341662 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.341702 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 01:07:24.342002 15407 log.cc:824] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:24.342231 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:24.342106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:39669)
0926 01:07:24.342183 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.342213 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 01:07:24.342360 15515 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:07:24.342479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 15384
I20260926 01:07:24.342712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35427
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45651
--webserver_port=39855
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:07:24.345955 15407 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:07:24.346220 15407 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Bootstrap complete.
I20260926 01:07:24.347075 15407 ts_tablet_manager.cc:1397] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20260926 01:07:24.348344 15407 raft_consensus.cc:348] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:24.348608 15407 raft_consensus.cc:729] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183c053291a0441b9b59751807b2d040, State: Initialized, Role: FOLLOWER
I20260926 01:07:24.348727 15407 consensus_queue.cc:260] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:24.348976 15407 ts_tablet_manager.cc:1428] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 01:07:24.349301 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:24.349187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:59957)
0926 01:07:24.349241 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.349273 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 01:07:24.349629 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:24.349526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:34877)
0926 01:07:24.349588 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.349612 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 01:07:24.429683 15519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:24.429932 15519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:24.429970 15519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:24.431452 15519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:24.431519 15519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:24.431561 15519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:07:24.433262 15519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39855
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:24.433542 15519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:24.433779 15519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:24.439634 15528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:24.439634 15526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:24.439635 15525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:24.440274 15519 server_base.cc:1034] running on GCE node
I20260926 01:07:24.440707 15519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:24.441203 15519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:24.442456 15519 hybrid_clock.cc:648] HybridClock initialized: now 1790384844442414 us; error 57 us; skew 500 ppm
Sep 26 01:07:24 dist-test-slave-l73v krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:24.445395 15519 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:07:24.445937 15519 webserver.cc:460] Webserver started at http://127.0.98.3:39855/ using document root <none> and password file <none>
I20260926 01:07:24.446161 15519 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:24.446331 15519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:24.448017 15519 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:07:24.448943 15535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:24.449169 15519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:07:24.449265 15519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "8353cc86dfca458294fac6c8c1ccba52"
format_stamp: "Formatted at 2026-09-26 01:07:22 on dist-test-slave-l73v"
server_key: "bd91811126faa0c48206a7f097234c76"
server_key_iv: "a03d3b312e721bf15e482ee454bc6f50"
server_key_version: "encryptionkey@0"
I20260926 01:07:24.449635 15519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:24.458285 15519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:24.460561 15519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:24.460750 15519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:24.460991 15519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:24.461313 15519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:24.461349 15519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:24.461380 15519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:24.461403 15519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:24.468885 15519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35427
I20260926 01:07:24.469043 15648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35427 every 8 connection(s)
I20260926 01:07:24.469353 15519 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:07:24.470721 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:24.470165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:58409)
0926 01:07:24.470414 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.470566 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
W20260926 01:07:24.471266 15649 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45651 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45651: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
I20260926 01:07:24.471665 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:24.471500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:45193)
0926 01:07:24.471605 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.471640 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 01:07:24.472096 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:24.471968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:33835)
0926 01:07:24.472038 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:24.472074 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 01:07:24.472323 15649 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:07:24.478314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 15519
I20260926 01:07:25.282351 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:25.282067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:50649)
0926 01:07:25.282220 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:25.282285 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260926 01:07:25.350492 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:25.350168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:38669)
0926 01:07:25.350331 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:25.350410 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260926 01:07:25.472999 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:25.472722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:36931)
0926 01:07:25.472885 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:25.472929 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260926 01:07:25.571621 15656 raft_consensus.cc:482] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:07:25.571756 15656 raft_consensus.cc:504] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:25.572110 15656 leader_election.cc:288] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 183c053291a0441b9b59751807b2d040 (127.0.98.2:41135), 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427)
Sep 26 01:07:25 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 26 01:07:25 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260926 01:07:25.582907 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:25.572505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33165 (local address 127.0.98.2:41135)
0926 01:07:25.572637 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:25.572650 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:25.572671 (+ 21us) server_negotiation.cc:407] Connection header received
0926 01:07:25.572857 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:25.572859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:25.572910 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:25.573041 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:25.574123 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:25.575242 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:25.576234 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:25.576632 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:25.576735 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:25.579474 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:25.579478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:25.579479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:25.579489 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:25.581176 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:25.581713 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:25.581716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:25.581717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:25.581824 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:25.582029 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:25.582030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:25.582030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:25.582279 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:25.582358 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:25.582664 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 01:07:25.582799 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260926 01:07:25.582929 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:25.572471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48499 (local address 127.0.98.3:35427)
0926 01:07:25.572648 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:25.572657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:25.572682 (+ 25us) server_negotiation.cc:407] Connection header received
0926 01:07:25.573402 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:25.573414 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:25.573479 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:25.573626 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:25.574089 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:25.575157 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:25.576221 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:25.576634 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:25.576735 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:25.579471 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:25.579476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:25.579477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:25.579489 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:25.581219 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:25.581697 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:25.581701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:25.581701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:25.581791 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:25.582026 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:25.582027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:25.582028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:25.582280 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:25.582354 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:25.582672 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 01:07:25.582809 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":56,"threads_started":1}
I20260926 01:07:25.582917 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:25.572351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41135 (local address 127.0.98.1:33165)
0926 01:07:25.572488 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:25.572498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:25.572556 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:25.573211 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:25.573214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:25.573778 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:25.573968 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:25.573974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:25.575393 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:25.575394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:25.576125 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:25.576127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:25.576718 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:25.576720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:25.576845 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:25.577457 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:25.577466 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:25.579067 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:25.581273 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:25.581284 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:25.581287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:25.581622 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:25.581892 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:25.581893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:25.581893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:25.581968 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:25.582346 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:25.582352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:25.582500 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 01:07:25.582592 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 01:07:25.582809 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":88,"thread_start_us":76,"threads_started":1}
I20260926 01:07:25.583004 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:25.572318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35427 (local address 127.0.98.1:48499)
0926 01:07:25.572367 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:25.572380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:25.572445 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:25.573627 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:25.573629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:25.573820 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:25.573968 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:25.573974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:25.575286 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:25.575288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:25.576125 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:25.576128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:25.576719 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:25.576720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:25.576845 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:25.577464 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:25.577468 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:25.579360 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:25.581326 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:25.581327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:25.581327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:25.581621 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:25.581860 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:25.581860 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:25.581861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:25.581967 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:25.582345 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:25.582352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:25.582500 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 01:07:25.582593 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 01:07:25.582887 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":142}
I20260926 01:07:25.583619 15469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e9acd2681147d6bfbbc65cf7aa544d" candidate_uuid: "0431139c48aa4ca88c3246a73f228cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "183c053291a0441b9b59751807b2d040" is_pre_election: true
I20260926 01:07:25.583788 15469 raft_consensus.cc:2454] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0431139c48aa4ca88c3246a73f228cce in term 1.
W20260926 01:07:25.583917 15268 leader_election.cc:341] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427): Not found: Tablet not found: e6e9acd2681147d6bfbbc65cf7aa544d
I20260926 01:07:25.584206 15269 leader_election.cc:302] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0431139c48aa4ca88c3246a73f228cce, 183c053291a0441b9b59751807b2d040; no voters: 8353cc86dfca458294fac6c8c1ccba52
I20260926 01:07:25.584385 15656 raft_consensus.cc:2781] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:07:25.584455 15656 raft_consensus.cc:482] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:07:25.584478 15656 raft_consensus.cc:3037] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:07:25.585774 15656 raft_consensus.cc:504] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:25.585922 15656 leader_election.cc:288] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [CANDIDATE]: Term 2 election: Requested vote from peers 183c053291a0441b9b59751807b2d040 (127.0.98.2:41135), 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427)
I20260926 01:07:25.586154 15469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e9acd2681147d6bfbbc65cf7aa544d" candidate_uuid: "0431139c48aa4ca88c3246a73f228cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "183c053291a0441b9b59751807b2d040"
I20260926 01:07:25.586247 15469 raft_consensus.cc:3037] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 1 FOLLOWER]: Advancing to term 2
W20260926 01:07:25.586357 15268 leader_election.cc:341] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427): Not found: Tablet not found: e6e9acd2681147d6bfbbc65cf7aa544d
I20260926 01:07:25.587431 15469 raft_consensus.cc:2454] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0431139c48aa4ca88c3246a73f228cce in term 2.
I20260926 01:07:25.587651 15269 leader_election.cc:302] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0431139c48aa4ca88c3246a73f228cce, 183c053291a0441b9b59751807b2d040; no voters: 8353cc86dfca458294fac6c8c1ccba52
I20260926 01:07:25.587741 15656 raft_consensus.cc:2781] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:07:25.587850 15656 raft_consensus.cc:686] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [term 2 LEADER]: Becoming Leader. State: Replica: 0431139c48aa4ca88c3246a73f228cce, State: Running, Role: LEADER
I20260926 01:07:25.587934 15656 consensus_queue.cc:237] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:25.588155 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:25.588046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:59569)
0926 01:07:25.588087 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:25.588125 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 01:07:26.178123 15268 consensus_peers.cc:487] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce -> Peer 8353cc86dfca458294fac6c8c1ccba52 (127.0.98.3:35427): Couldn't send request to peer 8353cc86dfca458294fac6c8c1ccba52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6e9acd2681147d6bfbbc65cf7aa544d. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:07:26.187014 15469 raft_consensus.cc:1261] T e6e9acd2681147d6bfbbc65cf7aa544d P 183c053291a0441b9b59751807b2d040 [term 2 FOLLOWER]: Refusing update from remote peer 0431139c48aa4ca88c3246a73f228cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 01:07:26.187366 15662 consensus_queue.cc:1035] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 01:07:26.188262 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:26.188097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:51881)
0926 01:07:26.188180 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:26.188220 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":41,"threads_started":1}
I20260926 01:07:26.189075 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:26.188907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:58741)
0926 01:07:26.188998 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:26.189035 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":52,"threads_started":1}
I20260926 01:07:26.189152 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:26.189055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:42701)
0926 01:07:26.189096 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:26.189124 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 01:07:26.473852 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:26.473611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:49627)
0926 01:07:26.473747 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:26.473804 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260926 01:07:26.616525 15671 ts_tablet_manager.cc:927] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Initiating tablet copy from peer 0431139c48aa4ca88c3246a73f228cce (127.0.98.1:38111)
I20260926 01:07:26.616784 15671 tablet_copy_client.cc:316] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:38111
Sep 26 01:07:26 dist-test-slave-l73v krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:07:26.625321 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:26.617079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.98.3:41781)
0926 01:07:26.617155 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:26.617175 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:26.617271 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:26.617596 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:26.617602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:26.617869 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:26.617987 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:26.617990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:26.619068 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:26.619069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:26.619720 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:26.619723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:26.620273 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:26.620274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:26.620352 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:26.620865 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:26.620870 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:26.622287 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:26.624214 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:26.624223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:26.624224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:26.624473 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:26.624680 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:26.624681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:26.624681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:26.624732 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:26.625007 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:26.625011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:26.625061 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 01:07:26.625126 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 01:07:26.625224 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 01:07:26.625396 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:26.617247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41781 (local address 127.0.98.1:38111)
0926 01:07:26.617443 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:26.617448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:26.617456 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:07:26.617480 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:26.617481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:26.617495 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:26.617554 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:26.618103 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:26.618940 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:26.619873 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:26.620196 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:26.620252 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:26.622428 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:26.622441 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:26.622441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:26.622451 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:26.624111 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:26.624556 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:26.624559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:26.624559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:26.624625 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:26.624773 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:26.624773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:26.624774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:26.624957 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:26.625018 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:26.625233 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 01:07:26.625312 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":110,"threads_started":1}
I20260926 01:07:26.625911 15354 tablet_copy_service.cc:140] P 0431139c48aa4ca88c3246a73f228cce: Received BeginTabletCopySession request for tablet e6e9acd2681147d6bfbbc65cf7aa544d from peer 8353cc86dfca458294fac6c8c1ccba52 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41781)
I20260926 01:07:26.625979 15354 tablet_copy_service.cc:161] P 0431139c48aa4ca88c3246a73f228cce: Beginning new tablet copy session on tablet e6e9acd2681147d6bfbbc65cf7aa544d from peer 8353cc86dfca458294fac6c8c1ccba52 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41781: session id = 8353cc86dfca458294fac6c8c1ccba52-e6e9acd2681147d6bfbbc65cf7aa544d
I20260926 01:07:26.626523 15354 tablet_copy_source_session.cc:205] T e6e9acd2681147d6bfbbc65cf7aa544d P 0431139c48aa4ca88c3246a73f228cce: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 01:07:26.627533 15671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e9acd2681147d6bfbbc65cf7aa544d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:26.630167 15671 tablet_copy_client.cc:791] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: tablet copy: Starting download of 0 data blocks...
I20260926 01:07:26.630303 15671 tablet_copy_client.cc:655] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: tablet copy: Starting download of 1 WAL segments...
I20260926 01:07:26.631170 15671 tablet_copy_client.cc:531] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 01:07:26.631848 15671 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Bootstrap starting.
I20260926 01:07:26.633154 15671 log.cc:824] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:26.635479 15671 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:07:26.635677 15671 tablet_bootstrap.cc:492] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Bootstrap complete.
I20260926 01:07:26.635792 15671 ts_tablet_manager.cc:1397] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 01:07:26.636751 15671 raft_consensus.cc:348] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:26.636922 15671 raft_consensus.cc:729] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8353cc86dfca458294fac6c8c1ccba52, State: Initialized, Role: FOLLOWER
I20260926 01:07:26.637042 15671 consensus_queue.cc:260] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183c053291a0441b9b59751807b2d040" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41135 } } peers { permanent_uuid: "8353cc86dfca458294fac6c8c1ccba52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35427 } } peers { permanent_uuid: "0431139c48aa4ca88c3246a73f228cce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260926 01:07:26.637286 15671 ts_tablet_manager.cc:1428] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260926 01:07:26.637717 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:26.637566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:57179)
0926 01:07:26.637643 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:26.637684 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 01:07:26.637755 15354 tablet_copy_service.cc:342] P 0431139c48aa4ca88c3246a73f228cce: Request end of tablet copy session 8353cc86dfca458294fac6c8c1ccba52-e6e9acd2681147d6bfbbc65cf7aa544d received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41781
I20260926 01:07:26.637877 15354 tablet_copy_service.cc:433] P 0431139c48aa4ca88c3246a73f228cce: ending tablet copy session 8353cc86dfca458294fac6c8c1ccba52-e6e9acd2681147d6bfbbc65cf7aa544d on tablet e6e9acd2681147d6bfbbc65cf7aa544d with peer 8353cc86dfca458294fac6c8c1ccba52
I20260926 01:07:27.189772 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:27.189574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:47189)
0926 01:07:27.189690 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:27.189720 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":45,"threads_started":1}
I20260926 01:07:27.189772 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:27.189566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:49049)
0926 01:07:27.189681 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:27.189721 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260926 01:07:27.197842 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:27.197683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:43361)
0926 01:07:27.197762 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:27.197804 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 01:07:27.197834 15603 raft_consensus.cc:1203] T e6e9acd2681147d6bfbbc65cf7aa544d P 8353cc86dfca458294fac6c8c1ccba52 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 01:07:27.198480 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:27.198258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:37153)
0926 01:07:27.198373 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:27.198437 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260926 01:07:28.190614 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:28.190364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:39443)
0926 01:07:28.190523 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:28.190566 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260926 01:07:28.198567 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:28.198320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:38515)
0926 01:07:28.198477 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:28.198512 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260926 01:07:28.199083 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:28.198886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:46385)
0926 01:07:28.199009 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:28.199045 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
I20260926 01:07:29.191494 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:29.191246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:39043)
0926 01:07:29.191402 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:29.191448 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260926 01:07:29.199262 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:29.199069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:43083)
0926 01:07:29.199187 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:29.199218 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260926 01:07:29.199694 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:29.199597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:44047)
0926 01:07:29.199643 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:29.199669 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:07:29.199708 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:29.199523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:49741)
0926 01:07:29.199635 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:29.199671 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260926 01:07:30.192579 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:30.192277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:54145)
0926 01:07:30.192488 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:30.192537 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260926 01:07:30.200541 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:30.200246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:48511)
0926 01:07:30.200436 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:30.200485 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":114,"threads_started":1}
I20260926 01:07:30.200541 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:30.200246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:36481)
0926 01:07:30.200436 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:30.200486 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
I20260926 01:07:31.193689 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:31.193401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:44013)
0926 01:07:31.193602 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:31.193645 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":107,"threads_started":1}
I20260926 01:07:31.201331 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:31.201099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:50219)
0926 01:07:31.201229 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:31.201274 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260926 01:07:31.201333 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:31.201098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:44043)
0926 01:07:31.201255 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:31.201287 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260926 01:07:32.194609 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:32.194292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:46329)
0926 01:07:32.194520 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:32.194560 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":60,"threads_started":1}
I20260926 01:07:32.202085 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:32.201842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:58535)
0926 01:07:32.201990 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:32.202043 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260926 01:07:32.202072 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:32.201839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:60435)
0926 01:07:32.201977 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:32.202020 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260926 01:07:33.195605 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:33.195351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:34663)
0926 01:07:33.195511 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:33.195558 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260926 01:07:33.202807 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:33.202562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:39859)
0926 01:07:33.202725 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:33.202761 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260926 01:07:33.202807 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:33.202562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:54383)
0926 01:07:33.202717 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:33.202761 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260926 01:07:34.196429 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:34.196186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:46047)
0926 01:07:34.196333 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:34.196380 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260926 01:07:34.203552 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:34.203315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:54495)
0926 01:07:34.203453 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:34.203487 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260926 01:07:34.203547 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:34.203315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:34007)
0926 01:07:34.203451 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:34.203487 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260926 01:07:35.197317 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:35.197039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:47393)
0926 01:07:35.197217 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:35.197269 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260926 01:07:35.204319 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:35.204032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:54659)
0926 01:07:35.204212 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:35.204259 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260926 01:07:35.204312 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:35.204032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:60873)
0926 01:07:35.204212 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:35.204258 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260926 01:07:36.198241 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:36.197900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:36557)
0926 01:07:36.198081 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:36.198131 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
I20260926 01:07:36.205101 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:36.204867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:57407)
0926 01:07:36.204999 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:36.205043 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260926 01:07:36.205135 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:36.204846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:33989)
0926 01:07:36.204988 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:36.205079 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260926 01:07:37.199120 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:37.198864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:44969)
0926 01:07:37.199014 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:37.199073 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260926 01:07:37.205853 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:37.205630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:40003)
0926 01:07:37.205765 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:37.205805 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260926 01:07:37.205885 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:37.205630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:56057)
0926 01:07:37.205809 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:37.205844 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
I20260926 01:07:38.199837 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:38.199600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:41537)
0926 01:07:38.199752 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:38.199790 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260926 01:07:38.206605 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:38.206349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:57963)
0926 01:07:38.206521 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:38.206557 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260926 01:07:38.206609 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:38.206350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:54845)
0926 01:07:38.206528 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:38.206568 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":139,"threads_started":1}
I20260926 01:07:39.200726 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:39.200419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:41987)
0926 01:07:39.200628 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:39.200678 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260926 01:07:39.207355 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:39.207115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:60583)
0926 01:07:39.207265 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:39.207307 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260926 01:07:39.207386 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:39.207163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:36407)
0926 01:07:39.207305 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:39.207344 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260926 01:07:39.481156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 15221
I20260926 01:07:39.485642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 15384
I20260926 01:07:39.490242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 15519
2026-09-26T01:07:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18254 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:07:39 dist-test-slave-l73v krb5kdc[15720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:07:39 dist-test-slave-l73v krb5kdc[15720](info): set up 2 sockets
Sep 26 01:07:39 dist-test-slave-l73v krb5kdc[15720](info): commencing operation
krb5kdc: starting...
W20260926 01:07:40.462363 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:07:40 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:07:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:07:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:07:40.589435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:40.670305 15733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:40.670567 15733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:40.670601 15733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:40.671929 15733 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 01:07:40.671976 15733 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 01:07:40.671993 15733 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 01:07:40.672008 15733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:07:40.672025 15733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:40.672049 15733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:40.672073 15733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:07:40.672091 15733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:07:40.673738 15733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:40.674005 15733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:40.674234 15733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:40.680053 15738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:40.680135 15739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:40.680178 15733 server_base.cc:1034] running on GCE node
W20260926 01:07:40.680135 15741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:40.680615 15733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:40.680915 15733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:40.682101 15733 hybrid_clock.cc:648] HybridClock initialized: now 1790384860682087 us; error 31 us; skew 500 ppm
Sep 26 01:07:40 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:40.685022 15733 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:07:40.685505 15733 webserver.cc:460] Webserver started at http://127.0.98.62:46585/ using document root <none> and password file <none>
I20260926 01:07:40.685726 15733 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:40.685777 15733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:40.685904 15733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:40.687081 15733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "c6bb3e8b1017410b9603e5c60e4e0379"
format_stamp: "Formatted at 2026-09-26 01:07:40 on dist-test-slave-l73v"
server_key: "e347a2b1bd63e45c0cf22e70c021dfbd"
server_key_iv: "209e4277d8bff1af3c407ebeccdbab31"
server_key_version: "encryptionkey@0"
I20260926 01:07:40.687419 15733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6bb3e8b1017410b9603e5c60e4e0379"
format_stamp: "Formatted at 2026-09-26 01:07:40 on dist-test-slave-l73v"
server_key: "e347a2b1bd63e45c0cf22e70c021dfbd"
server_key_iv: "209e4277d8bff1af3c407ebeccdbab31"
server_key_version: "encryptionkey@0"
I20260926 01:07:40.688789 15733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:07:40.689563 15748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:40.689791 15733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:40.689898 15733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "c6bb3e8b1017410b9603e5c60e4e0379"
format_stamp: "Formatted at 2026-09-26 01:07:40 on dist-test-slave-l73v"
server_key: "e347a2b1bd63e45c0cf22e70c021dfbd"
server_key_iv: "209e4277d8bff1af3c407ebeccdbab31"
server_key_version: "encryptionkey@0"
I20260926 01:07:40.689968 15733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:40.722434 15733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:40.724928 15733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:40.725098 15733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:40.729526 15733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36245
I20260926 01:07:40.729619 15800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36245 every 8 connection(s)
I20260926 01:07:40.729961 15733 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:07:40.730957 15801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:40.733560 15801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Bootstrap starting.
I20260926 01:07:40.734191 15801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:40.734508 15801 log.cc:824] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:40.734882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 15733
I20260926 01:07:40.734984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:07:40.735263 392 external_mini_cluster.cc:1428] Setting key c96d889b9749ce7626d8045aea0bf597
I20260926 01:07:40.735589 15801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: No bootstrap required, opened a new log
I20260926 01:07:40.737620 15801 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:07:40.737787 15801 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:40.737828 15801 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bb3e8b1017410b9603e5c60e4e0379, State: Initialized, Role: FOLLOWER
I20260926 01:07:40.737986 15801 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:07:40.738067 15801 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:07:40.738101 15801 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:07:40.738152 15801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:40.739147 15801 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:07:40.739259 15801 leader_election.cc:302] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6bb3e8b1017410b9603e5c60e4e0379; no voters:
I20260926 01:07:40.739442 15801 leader_election.cc:288] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:07:40.739502 15806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:07:40.739691 15801 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:07:40.739706 15806 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 1 LEADER]: Becoming Leader. State: Replica: c6bb3e8b1017410b9603e5c60e4e0379, State: Running, Role: LEADER
I20260926 01:07:40.739863 15806 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:07:40.740268 15808 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6bb3e8b1017410b9603e5c60e4e0379. Latest consensus state: current_term: 1 leader_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } } }
I20260926 01:07:40.740366 15808 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:40.740319 15807 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } } }
I20260926 01:07:40.740466 15807 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: This master's current role is: LEADER
I20260926 01:07:40.740679 15810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:07:40.741214 15810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:07:40.742725 15823 mvcc.cc:205] Tried to move back new op lower bound from 7333416389600256000 to 7333416389590982656. Current Snapshot: MvccSnapshot[applied={T|T < 7333416389600256000}]
Sep 26 01:07:40 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:40.744892 15810 catalog_manager.cc:1347] Generated new cluster ID: b37ba87c7b6a4b84bc3de1ae5e622430
I20260926 01:07:40.744946 15810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:07:40.748400 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:40.735797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.0.98.62:36245)
0926 01:07:40.736143 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:40.736151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:40.736177 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:07:40.737608 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:40.737618 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:40.737683 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:40.737879 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:40.738634 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:40.739735 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:40.740779 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:40.741391 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:40.742708 (+ 1317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:40.744187 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:40.744191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:40.744192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:40.744204 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:40.746323 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:40.746829 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:40.746834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:40.746835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:40.747013 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:40.747208 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:40.747209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:40.747210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:40.747598 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:40.747752 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:40.747850 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 01:07:40.748018 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":236,"threads_started":1}
I20260926 01:07:40.750312 15810 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:07:40.751147 15810 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:07:40.756377 15810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Generated new TSK 0
I20260926 01:07:40.756603 15810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:07:40.790532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36245
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:40.879141 15827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:40.879374 15827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:40.879400 15827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:40.880738 15827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:40.880793 15827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:40.880824 15827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:07:40.882552 15827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:40.882833 15827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:40.883095 15827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:40.888373 15833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:40.888484 15832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:40.888373 15835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:40.888664 15827 server_base.cc:1034] running on GCE node
I20260926 01:07:40.888932 15827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:40.889205 15827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:40.890422 15827 hybrid_clock.cc:648] HybridClock initialized: now 1790384860890404 us; error 31 us; skew 500 ppm
Sep 26 01:07:40 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:40.893186 15827 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:07:40.893951 15827 webserver.cc:460] Webserver started at http://127.0.98.1:42625/ using document root <none> and password file <none>
I20260926 01:07:40.894189 15827 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:40.894236 15827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:40.894351 15827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:40.895514 15827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "80a7d651276340fca1bfa4b4157c5703"
format_stamp: "Formatted at 2026-09-26 01:07:40 on dist-test-slave-l73v"
server_key: "09f24bf1a50c793ba9f5492888e691f9"
server_key_iv: "54ddc8bcb7f034060bce5faa6fdcca40"
server_key_version: "encryptionkey@0"
I20260926 01:07:40.895825 15827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "80a7d651276340fca1bfa4b4157c5703"
format_stamp: "Formatted at 2026-09-26 01:07:40 on dist-test-slave-l73v"
server_key: "09f24bf1a50c793ba9f5492888e691f9"
server_key_iv: "54ddc8bcb7f034060bce5faa6fdcca40"
server_key_version: "encryptionkey@0"
I20260926 01:07:40.897154 15827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:07:40.898025 15842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:40.898236 15827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:07:40.898339 15827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "80a7d651276340fca1bfa4b4157c5703"
format_stamp: "Formatted at 2026-09-26 01:07:40 on dist-test-slave-l73v"
server_key: "09f24bf1a50c793ba9f5492888e691f9"
server_key_iv: "54ddc8bcb7f034060bce5faa6fdcca40"
server_key_version: "encryptionkey@0"
I20260926 01:07:40.898440 15827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:40.909883 15827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:40.912179 15827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:40.912313 15827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:40.912514 15827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:40.912802 15827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:40.912837 15827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:40.912863 15827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:40.912883 15827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:40.919088 15827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39591
I20260926 01:07:40.919160 15955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39591 every 8 connection(s)
I20260926 01:07:40.919480 15827 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 01:07:40 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:40.925911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 15827
I20260926 01:07:40.926007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:07:40.926189 392 external_mini_cluster.cc:1428] Setting key 23d861db8f26531183df6302a2ccbbd3
I20260926 01:07:40.929088 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:40.920215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58261 (local address 127.0.98.62:36245)
0926 01:07:40.920265 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:40.920267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:40.920460 (+ 193us) server_negotiation.cc:407] Connection header received
0926 01:07:40.920998 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:40.920999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:40.921014 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:40.921072 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:40.922021 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:40.922571 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:40.923654 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:40.923905 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:40.923985 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:40.926351 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:40.926353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:40.926354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:40.926366 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:40.927747 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:40.928151 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:40.928152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:40.928152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:40.928207 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:40.928448 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:40.928449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:40.928449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:40.928566 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:40.928622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:40.928902 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 01:07:40.928992 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:07:40.929268 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:40.920184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:58261)
0926 01:07:40.920343 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:40.920388 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:40.920849 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:40.921131 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:40.921135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:40.921707 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:40.921915 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:40.921920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:40.922715 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:40.922716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:40.923554 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:40.923556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:40.924035 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:40.924036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:40.924165 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:40.924713 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:40.924722 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:40.926219 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:40.927830 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:40.927841 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:40.927843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:40.928082 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:40.928265 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:40.928266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:40.928266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:40.928391 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:40.928620 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:40.928628 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:40.928777 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 01:07:40.928874 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 01:07:40.929046 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260926 01:07:40.929733 15956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36245
I20260926 01:07:40.929828 15956 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:40.930002 15956 heartbeater.cc:511] Master 127.0.98.62:36245 requested a full tablet report, sending...
I20260926 01:07:40.930580 15765 ts_manager.cc:194] Registered new tserver with Master: 80a7d651276340fca1bfa4b4157c5703 (127.0.98.1:39591)
I20260926 01:07:40.931483 15765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58261
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:07:40.954087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36245
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:41.035998 15961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:41.036240 15961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:41.036279 15961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:41.037580 15961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:41.037639 15961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:41.037678 15961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:07:41.039324 15961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:41.039680 15961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:41.039893 15961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:41.045230 15967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:41.045230 15966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:41.045296 15969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:41.045310 15961 server_base.cc:1034] running on GCE node
I20260926 01:07:41.045710 15961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:41.045979 15961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:41.047124 15961 hybrid_clock.cc:648] HybridClock initialized: now 1790384861047105 us; error 35 us; skew 500 ppm
Sep 26 01:07:41 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:41.049523 15961 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:07:41.049995 15961 webserver.cc:460] Webserver started at http://127.0.98.2:37655/ using document root <none> and password file <none>
I20260926 01:07:41.050207 15961 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:41.050254 15961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:41.050421 15961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:41.051461 15961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "619c729ef81341359bf4996ceba10ffb"
format_stamp: "Formatted at 2026-09-26 01:07:41 on dist-test-slave-l73v"
server_key: "6317c357fea140e25b0fadc8bb72bdca"
server_key_iv: "a27c51b0421b614a9fa35d62ea1196fd"
server_key_version: "encryptionkey@0"
I20260926 01:07:41.051780 15961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "619c729ef81341359bf4996ceba10ffb"
format_stamp: "Formatted at 2026-09-26 01:07:41 on dist-test-slave-l73v"
server_key: "6317c357fea140e25b0fadc8bb72bdca"
server_key_iv: "a27c51b0421b614a9fa35d62ea1196fd"
server_key_version: "encryptionkey@0"
I20260926 01:07:41.053058 15961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:07:41.053802 15976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:41.053994 15961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:07:41.054076 15961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "619c729ef81341359bf4996ceba10ffb"
format_stamp: "Formatted at 2026-09-26 01:07:41 on dist-test-slave-l73v"
server_key: "6317c357fea140e25b0fadc8bb72bdca"
server_key_iv: "a27c51b0421b614a9fa35d62ea1196fd"
server_key_version: "encryptionkey@0"
I20260926 01:07:41.054143 15961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:41.067191 15961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:41.069177 15961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:41.069321 15961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:41.069561 15961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:41.069864 15961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:41.069897 15961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:41.069926 15961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:41.069948 15961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:41.076010 15961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44351
I20260926 01:07:41.076107 16089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44351 every 8 connection(s)
I20260926 01:07:41.076424 15961 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:07:41.078887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 15961
I20260926 01:07:41.078998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:07:41.079188 392 external_mini_cluster.cc:1428] Setting key 493de97dd48b6ac8712587e2915897e0
Sep 26 01:07:41 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:41.087147 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.077298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59913 (local address 127.0.98.62:36245)
0926 01:07:41.077398 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:41.077418 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:41.077569 (+ 151us) server_negotiation.cc:407] Connection header received
0926 01:07:41.078116 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:41.078117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:41.078131 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:41.078181 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:41.079399 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.079946 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.081290 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.081484 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.081548 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.084028 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:41.084031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:41.084031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:41.084041 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:41.085644 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:41.086118 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:41.086119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:41.086120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:41.086188 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:41.086462 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:41.086463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:41.086464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:41.086623 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:41.086694 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:41.086949 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 01:07:41.087047 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 01:07:41.087219 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.077223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:59913)
0926 01:07:41.077427 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:41.077477 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:41.078010 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:41.078258 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:41.078261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:41.078988 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:41.079292 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.079299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.080033 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.080034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.081190 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.081193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.081600 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.081601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.081769 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.082522 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:41.082534 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:41.083908 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:41.085744 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:41.085752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:41.085754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:41.086046 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:41.086251 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:41.086252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:41.086252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:41.086351 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:41.086682 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:41.086687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:41.086817 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 01:07:41.086900 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 01:07:41.087037 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260926 01:07:41.087739 16090 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36245
I20260926 01:07:41.087869 16090 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:41.088091 16090 heartbeater.cc:511] Master 127.0.98.62:36245 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:07:41.088516 15765 ts_manager.cc:194] Registered new tserver with Master: 619c729ef81341359bf4996ceba10ffb (127.0.98.2:44351)
I20260926 01:07:41.088897 15765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:07:41.106367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36245
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:07:41.191680 16095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:07:41.191915 16095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:07:41.191942 16095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:07:41.193269 16095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:07:41.193324 16095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:07:41.193352 16095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:07:41.194995 16095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:07:41.195250 16095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:07:41.195495 16095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:07:41.200798 16101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:41.200841 16103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:07:41.200851 16100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:07:41.201128 16095 server_base.cc:1034] running on GCE node
I20260926 01:07:41.201351 16095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:07:41.201597 16095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:07:41.202800 16095 hybrid_clock.cc:648] HybridClock initialized: now 1790384861202807 us; error 55 us; skew 500 ppm
Sep 26 01:07:41 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:07:41.205504 16095 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:07:41.205996 16095 webserver.cc:460] Webserver started at http://127.0.98.3:41197/ using document root <none> and password file <none>
I20260926 01:07:41.206225 16095 fs_manager.cc:359] Metadata directory not provided
I20260926 01:07:41.206276 16095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:07:41.206436 16095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:07:41.207561 16095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "92c995d3125d4be2838836de812a6abc"
format_stamp: "Formatted at 2026-09-26 01:07:41 on dist-test-slave-l73v"
server_key: "a358b6070c0179c1edb76ede71068eb1"
server_key_iv: "ebcb02637b2da8cce98bc87823f72684"
server_key_version: "encryptionkey@0"
I20260926 01:07:41.207916 16095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "92c995d3125d4be2838836de812a6abc"
format_stamp: "Formatted at 2026-09-26 01:07:41 on dist-test-slave-l73v"
server_key: "a358b6070c0179c1edb76ede71068eb1"
server_key_iv: "ebcb02637b2da8cce98bc87823f72684"
server_key_version: "encryptionkey@0"
I20260926 01:07:41.209326 16095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:07:41.210124 16110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:41.210398 16095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:07:41.210564 16095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "92c995d3125d4be2838836de812a6abc"
format_stamp: "Formatted at 2026-09-26 01:07:41 on dist-test-slave-l73v"
server_key: "a358b6070c0179c1edb76ede71068eb1"
server_key_iv: "ebcb02637b2da8cce98bc87823f72684"
server_key_version: "encryptionkey@0"
I20260926 01:07:41.210650 16095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:07:41.217948 16095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:07:41.220176 16095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:07:41.220293 16095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:07:41.220492 16095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:07:41.220745 16095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:07:41.220773 16095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:41.220794 16095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:07:41.220810 16095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:07:41.227119 16095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32815
I20260926 01:07:41.227161 16223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32815 every 8 connection(s)
I20260926 01:07:41.227494 16095 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:07:41.231398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 16095
I20260926 01:07:41.231503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:07:41.231669 392 external_mini_cluster.cc:1428] Setting key 89729c2d262b53ebc79d44f45b2ca49b
Sep 26 01:07:41 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:07:41.237557 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.228234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38973 (local address 127.0.98.62:36245)
0926 01:07:41.228315 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:41.228317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:41.228523 (+ 206us) server_negotiation.cc:407] Connection header received
0926 01:07:41.229503 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:41.229505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:41.229515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:41.229574 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:41.230264 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.230888 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.232067 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.232253 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.232309 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.234822 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:41.234824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:41.234825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:41.234835 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:41.236176 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:41.236594 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:41.236596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:41.236596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:41.236649 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:41.236898 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:41.236899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:41.236899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:41.237020 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:41.237077 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:41.237377 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 01:07:41.237463 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 01:07:41.237756 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.228206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:38973)
0926 01:07:41.228402 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:41.228439 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:41.229375 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:41.229652 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:41.229655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:41.229954 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:07:41.230149 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.230154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.230994 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.230995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.231934 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.231937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.232359 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.232361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.232475 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.233014 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:07:41.233023 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:07:41.234714 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:07:41.236266 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:41.236276 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:41.236279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:41.236511 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:41.236709 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:07:41.236710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:07:41.236710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:07:41.236837 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:07:41.237076 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:07:41.237083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:07:41.237248 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 01:07:41.237342 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 01:07:41.237516 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260926 01:07:41.238279 16224 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36245
I20260926 01:07:41.238374 16224 heartbeater.cc:464] Registering TS with master...
I20260926 01:07:41.238574 16224 heartbeater.cc:511] Master 127.0.98.62:36245 requested a full tablet report, sending...
I20260926 01:07:41.238977 15765 ts_manager.cc:194] Registered new tserver with Master: 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815)
I20260926 01:07:41.239321 15765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38973
I20260926 01:07:41.243372 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:07:41.250495 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.244727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.0.98.62:36245)
0926 01:07:41.244789 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:41.244791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:41.244947 (+ 156us) server_negotiation.cc:407] Connection header received
0926 01:07:41.245022 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:41.245023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:41.245033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:41.245073 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:07:41.245642 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.246125 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.246880 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.247063 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.247117 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.248339 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:07:41.248341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:07:41.248341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:07:41.248349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:07:41.249359 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:41.249677 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:41.249677 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:41.249678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:41.249730 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:07:41.249889 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:07:41.249890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:07:41.249890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:07:41.250047 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:07:41.250137 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:41.250276 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 01:07:41.250361 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:07:41.252167 15765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:07:41.253695 15765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:07:41.260208 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.255806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.0.98.3:32815)
0926 01:07:41.256264 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:41.256269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:41.256278 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:07:41.256317 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:41.256319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:41.256334 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:41.256657 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:41.256730 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.257806 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.259009 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.259758 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.259836 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.259906 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:41.260074 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 01:07:41.260113 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":83,"threads_started":1}
I20260926 01:07:41.260233 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.255700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32815 (local address 127.0.0.1:60174)
0926 01:07:41.255987 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:41.255996 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:41.256034 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:41.256484 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:41.256485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:41.256488 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:41.256524 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.256525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.257884 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.257885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.258929 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.258932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.259843 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.259844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.259984 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.259987 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:07:41.260055 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 01:07:41.260125 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260926 01:07:41.260492 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.255807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.0.98.2:44351)
0926 01:07:41.256121 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:41.256126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:41.256135 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:07:41.256166 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:41.256168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:41.256180 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:41.256276 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:41.256482 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.257722 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.259049 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.259759 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.259836 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.259899 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:41.260360 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 01:07:41.260405 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":90,"threads_started":1}
I20260926 01:07:41.260234 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.255698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44351 (local address 127.0.0.1:43320)
0926 01:07:41.255874 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:41.255925 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:41.255991 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:41.256296 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:41.256299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:41.256301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:41.256378 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.256396 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.257825 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.257826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.258979 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.258980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.259842 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.259843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.259962 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.259979 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 01:07:41.260056 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 01:07:41.260125 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":84,"thread_start_us":88,"threads_started":1}
I20260926 01:07:41.261066 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.255917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39591 (local address 127.0.0.1:36274)
0926 01:07:41.256085 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:41.256096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:41.256146 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:41.256421 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:41.256422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:41.256424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:41.256486 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.256488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.257558 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.257559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.258976 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.258979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.260831 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.260832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.260894 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.260899 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:07:41.260953 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 01:07:41.261000 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":40,"threads_started":1}
I20260926 01:07:41.261077 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.256102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.0.98.1:39591)
0926 01:07:41.256277 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:41.256282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:41.256291 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:07:41.256323 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:41.256325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:41.256339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:41.256415 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:41.256537 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.257467 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.259941 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.260679 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.260745 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.260798 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:41.260980 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 01:07:41.261016 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260926 01:07:41.261089 16158 tablet_service.cc:1467] Processing CreateTablet for tablet cee4ec99f225490297b75ffe62ea281e (DEFAULT_TABLE table=test-table [id=ca172170244847969f1d1576ccfd4c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:41.261444 16158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee4ec99f225490297b75ffe62ea281e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:41.261718 15890 tablet_service.cc:1467] Processing CreateTablet for tablet cee4ec99f225490297b75ffe62ea281e (DEFAULT_TABLE table=test-table [id=ca172170244847969f1d1576ccfd4c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:41.262028 15890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee4ec99f225490297b75ffe62ea281e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:41.264046 16238 tablet_bootstrap.cc:492] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: Bootstrap starting.
I20260926 01:07:41.264408 16239 tablet_bootstrap.cc:492] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Bootstrap starting.
I20260926 01:07:41.264775 16238 tablet_bootstrap.cc:654] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:41.264964 16024 tablet_service.cc:1467] Processing CreateTablet for tablet cee4ec99f225490297b75ffe62ea281e (DEFAULT_TABLE table=test-table [id=ca172170244847969f1d1576ccfd4c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:07:41.265082 16239 tablet_bootstrap.cc:654] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:41.265261 16238 log.cc:824] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:41.265298 16024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee4ec99f225490297b75ffe62ea281e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:07:41.265635 16239 log.cc:824] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:41.265964 16238 tablet_bootstrap.cc:492] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: No bootstrap required, opened a new log
I20260926 01:07:41.266039 16238 ts_tablet_manager.cc:1397] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:07:41.266309 16239 tablet_bootstrap.cc:492] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: No bootstrap required, opened a new log
I20260926 01:07:41.266444 16239 ts_tablet_manager.cc:1397] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:07:41.267835 16242 tablet_bootstrap.cc:492] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: Bootstrap starting.
I20260926 01:07:41.268013 16238 raft_consensus.cc:348] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:07:41.268198 16238 raft_consensus.cc:374] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:41.268231 16238 raft_consensus.cc:729] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c995d3125d4be2838836de812a6abc, State: Initialized, Role: FOLLOWER
I20260926 01:07:41.268373 16238 consensus_queue.cc:260] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:07:41.268463 16242 tablet_bootstrap.cc:654] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: Neither blocks nor log segments found. Creating new log.
I20260926 01:07:41.268661 16238 ts_tablet_manager.cc:1428] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 01:07:41.268800 16224 heartbeater.cc:503] Master 127.0.98.62:36245 was elected leader, sending a full tablet report...
I20260926 01:07:41.268751 16239 raft_consensus.cc:348] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:07:41.268924 16239 raft_consensus.cc:374] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:41.268990 16239 raft_consensus.cc:729] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a7d651276340fca1bfa4b4157c5703, State: Initialized, Role: FOLLOWER
I20260926 01:07:41.268992 16242 log.cc:824] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:07:41.269114 16239 consensus_queue.cc:260] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:07:41.269356 16239 ts_tablet_manager.cc:1428] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:07:41.269421 15956 heartbeater.cc:503] Master 127.0.98.62:36245 was elected leader, sending a full tablet report...
I20260926 01:07:41.270144 16242 tablet_bootstrap.cc:492] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: No bootstrap required, opened a new log
I20260926 01:07:41.270211 16242 ts_tablet_manager.cc:1397] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:07:41.271723 16242 raft_consensus.cc:348] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:07:41.271862 16242 raft_consensus.cc:374] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:07:41.271883 16242 raft_consensus.cc:729] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619c729ef81341359bf4996ceba10ffb, State: Initialized, Role: FOLLOWER
I20260926 01:07:41.271989 16242 consensus_queue.cc:260] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:07:41.272225 16242 ts_tablet_manager.cc:1428] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:07:41.272275 16090 heartbeater.cc:503] Master 127.0.98.62:36245 was elected leader, sending a full tablet report...
W20260926 01:07:41.327014 16091 tablet.cc:2369] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:07:41.420266 15957 tablet.cc:2369] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:07:41.467799 16245 raft_consensus.cc:482] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:07:41.467929 16245 raft_consensus.cc:504] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:07:41.468276 16245 leader_election.cc:288] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 619c729ef81341359bf4996ceba10ffb (127.0.98.2:44351), 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815)
I20260926 01:07:41.471194 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.468517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50645 (local address 127.0.98.2:44351)
0926 01:07:41.468562 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:41.468564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:41.468678 (+ 114us) server_negotiation.cc:407] Connection header received
0926 01:07:41.468734 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:41.468735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:41.468747 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:41.468795 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:41.468982 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.469456 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.470530 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.470889 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.470949 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.470960 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:41.471092 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 01:07:41.471128 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:07:41.471197 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.468540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42467 (local address 127.0.98.3:32815)
0926 01:07:41.468589 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:41.468591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:41.468691 (+ 100us) server_negotiation.cc:407] Connection header received
0926 01:07:41.468742 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:41.468743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:41.468754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:41.468800 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:07:41.469013 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.469461 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.470533 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.470891 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.470953 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.470964 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:41.471092 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 01:07:41.471130 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:07:41.471220 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.468458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32815 (local address 127.0.98.1:42467)
0926 01:07:41.468614 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:41.468624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:41.468684 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:41.468859 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:41.468860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:41.468862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:41.468940 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.468941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.469536 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.469537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.470452 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.470455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.470963 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.470964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.471032 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.471035 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:07:41.471079 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:07:41.471120 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":44,"thread_start_us":79,"threads_started":1}
I20260926 01:07:41.471222 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.468454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44351 (local address 127.0.98.1:50645)
0926 01:07:41.468603 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:41.468613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:07:41.468677 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:07:41.468844 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:07:41.468845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:07:41.468847 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:07:41.468919 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.468920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.469540 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.469541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.470456 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:07:41.470459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.470968 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.470969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:07:41.471034 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.471037 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:07:41.471082 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:07:41.471127 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260926 01:07:41.471518 16178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee4ec99f225490297b75ffe62ea281e" candidate_uuid: "80a7d651276340fca1bfa4b4157c5703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c995d3125d4be2838836de812a6abc" is_pre_election: true
I20260926 01:07:41.471681 16178 raft_consensus.cc:2454] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a7d651276340fca1bfa4b4157c5703 in term 0.
I20260926 01:07:41.471748 16044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee4ec99f225490297b75ffe62ea281e" candidate_uuid: "80a7d651276340fca1bfa4b4157c5703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619c729ef81341359bf4996ceba10ffb" is_pre_election: true
I20260926 01:07:41.471908 16044 raft_consensus.cc:2454] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a7d651276340fca1bfa4b4157c5703 in term 0.
I20260926 01:07:41.471899 15844 leader_election.cc:302] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a7d651276340fca1bfa4b4157c5703, 92c995d3125d4be2838836de812a6abc; no voters:
I20260926 01:07:41.472018 16245 raft_consensus.cc:2781] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:07:41.472069 16245 raft_consensus.cc:482] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:07:41.472091 16245 raft_consensus.cc:3037] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:41.472786 16245 raft_consensus.cc:504] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:07:41.472918 16245 leader_election.cc:288] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [CANDIDATE]: Term 1 election: Requested vote from peers 619c729ef81341359bf4996ceba10ffb (127.0.98.2:44351), 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815)
I20260926 01:07:41.473083 16178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee4ec99f225490297b75ffe62ea281e" candidate_uuid: "80a7d651276340fca1bfa4b4157c5703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c995d3125d4be2838836de812a6abc"
I20260926 01:07:41.473116 16044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee4ec99f225490297b75ffe62ea281e" candidate_uuid: "80a7d651276340fca1bfa4b4157c5703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619c729ef81341359bf4996ceba10ffb"
I20260926 01:07:41.473153 16178 raft_consensus.cc:3037] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:41.473207 16044 raft_consensus.cc:3037] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:07:41.473958 16044 raft_consensus.cc:2454] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a7d651276340fca1bfa4b4157c5703 in term 1.
I20260926 01:07:41.473960 16178 raft_consensus.cc:2454] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a7d651276340fca1bfa4b4157c5703 in term 1.
I20260926 01:07:41.474150 15843 leader_election.cc:302] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619c729ef81341359bf4996ceba10ffb, 80a7d651276340fca1bfa4b4157c5703; no voters:
I20260926 01:07:41.474242 16245 raft_consensus.cc:2781] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:07:41.474403 16245 raft_consensus.cc:686] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 1 LEADER]: Becoming Leader. State: Replica: 80a7d651276340fca1bfa4b4157c5703, State: Running, Role: LEADER
I20260926 01:07:41.474512 16245 consensus_queue.cc:237] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:07:41.475405 15764 catalog_manager.cc:5495] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a7d651276340fca1bfa4b4157c5703 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "80a7d651276340fca1bfa4b4157c5703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } health_report { overall_health: HEALTHY } } }
W20260926 01:07:41.478029 16225 tablet.cc:2369] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:07:41.490795 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.488539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.0.98.1:39591)
0926 01:07:41.488603 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:07:41.488605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:07:41.488611 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:07:41.488645 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:07:41.488646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:07:41.488666 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:07:41.488724 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:07:41.488879 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.489357 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.490075 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:07:41.490274 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:07:41.490328 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:07:41.490493 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:07:41.490596 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:07:41.490670 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:07:41.490693 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 01:07:41.490713 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:07:41.492770 16178 raft_consensus.cc:1261] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc [term 1 FOLLOWER]: Refusing update from remote peer 80a7d651276340fca1bfa4b4157c5703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:07:41.492774 16044 raft_consensus.cc:1261] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Refusing update from remote peer 80a7d651276340fca1bfa4b4157c5703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:07:41.493104 16245 consensus_queue.cc:1035] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:07:41.493251 16249 consensus_queue.cc:1035] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:07:41.495662 16252 mvcc.cc:205] Tried to move back new op lower bound from 7333416392671178752 to 7333416392600211456. Current Snapshot: MvccSnapshot[applied={T|T < 7333416392671178752}]
I20260926 01:07:41.495956 16254 mvcc.cc:205] Tried to move back new op lower bound from 7333416392671178752 to 7333416392600211456. Current Snapshot: MvccSnapshot[applied={T|T < 7333416392671178752}]
I20260926 01:07:41.496320 16253 mvcc.cc:205] Tried to move back new op lower bound from 7333416392671178752 to 7333416392600211456. Current Snapshot: MvccSnapshot[applied={T|T < 7333416392671178752}]
I20260926 01:07:41.498812 15765 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:07:41.498926 15765 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:07:41.499729 15765 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Sending DeleteTablet for 3 replicas of tablet cee4ec99f225490297b75ffe62ea281e
I20260926 01:07:41.500073 15890 tablet_service.cc:1514] Processing DeleteTablet for tablet cee4ec99f225490297b75ffe62ea281e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36274
I20260926 01:07:41.500121 16024 tablet_service.cc:1514] Processing DeleteTablet for tablet cee4ec99f225490297b75ffe62ea281e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43320
I20260926 01:07:41.500118 16158 tablet_service.cc:1514] Processing DeleteTablet for tablet cee4ec99f225490297b75ffe62ea281e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60174
I20260926 01:07:41.500360 16262 tablet_replica.cc:330] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: stopping tablet replica
I20260926 01:07:41.500469 16262 raft_consensus.cc:2229] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:07:41.500473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 15827
I20260926 01:07:41.500607 16262 raft_consensus.cc:2258] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:41.500751 16264 tablet_replica.cc:330] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: stopping tablet replica
I20260926 01:07:41.500839 16264 raft_consensus.cc:2229] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:07:41.500982 16264 raft_consensus.cc:2258] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:41.501462 16264 ts_tablet_manager.cc:1905] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:07:41.503521 16264 ts_tablet_manager.cc:1918] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:07:41.503611 16264 log.cc:1196] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/cee4ec99f225490297b75ffe62ea281e
I20260926 01:07:41.503870 16264 ts_tablet_manager.cc:1939] T cee4ec99f225490297b75ffe62ea281e P 619c729ef81341359bf4996ceba10ffb: Deleting consensus metadata
I20260926 01:07:41.504338 15749 catalog_manager.cc:4841] TS 619c729ef81341359bf4996ceba10ffb (127.0.98.2:44351): tablet cee4ec99f225490297b75ffe62ea281e (table test-table [id=ca172170244847969f1d1576ccfd4c46]) successfully deleted
W20260926 01:07:41.505698 15749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39591 (error 108)
I20260926 01:07:41.505985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 15961
I20260926 01:07:41.506537 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:07:41.506056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39591 (local address 127.0.0.1:36294)
0926 01:07:41.506459 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 01:07:41.506503 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379}
I20260926 01:07:41.507296 16263 tablet_replica.cc:330] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: stopping tablet replica
I20260926 01:07:41.507436 16263 raft_consensus.cc:2229] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:07:41.507565 16263 raft_consensus.cc:2258] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:07:41.508051 16263 ts_tablet_manager.cc:1905] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:07:41.509931 16263 ts_tablet_manager.cc:1918] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:07:41.510015 16263 log.cc:1196] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/cee4ec99f225490297b75ffe62ea281e
I20260926 01:07:41.510283 16263 ts_tablet_manager.cc:1939] T cee4ec99f225490297b75ffe62ea281e P 92c995d3125d4be2838836de812a6abc: Deleting consensus metadata
W20260926 01:07:41.510543 15749 catalog_manager.cc:4584] TS 80a7d651276340fca1bfa4b4157c5703 (127.0.98.1:39591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cee4ec99f225490297b75ffe62ea281e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39591: connect: Connection refused (error 111)
I20260926 01:07:41.510819 15750 catalog_manager.cc:4841] TS 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815): tablet cee4ec99f225490297b75ffe62ea281e (table test-table [id=ca172170244847969f1d1576ccfd4c46]) successfully deleted
I20260926 01:07:41.511664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 16095
I20260926 01:07:41.516014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 15733
I20260926 01:08:01.520905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36245
--webserver_interface=127.0.98.62
--webserver_port=46585
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:01.604377 16266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:01.604624 16266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:01.604647 16266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:01.605901 16266 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 01:08:01.605953 16266 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 01:08:01.605966 16266 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 01:08:01.605976 16266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:01.605988 16266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:01.606000 16266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:01.606010 16266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:01.606021 16266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:01.607673 16266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:01.607910 16266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:01.608099 16266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:01.617110 16273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:01.617125 16272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:01.617111 16266 server_base.cc:1034] running on GCE node
W20260926 01:08:01.617127 16275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:01.617619 16266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:01.617983 16266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:01.619148 16266 hybrid_clock.cc:648] HybridClock initialized: now 1790384881619128 us; error 36 us; skew 500 ppm
Sep 26 01:08:01 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:01.625010 16266 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:08:01.625522 16266 webserver.cc:460] Webserver started at http://127.0.98.62:46585/ using document root <none> and password file <none>
I20260926 01:08:01.625803 16266 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:01.625862 16266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:01.628086 16266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:01.628973 16282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:01.629176 16266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:01.629269 16266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "c6bb3e8b1017410b9603e5c60e4e0379"
format_stamp: "Formatted at 2026-09-26 01:07:40 on dist-test-slave-l73v"
server_key: "e347a2b1bd63e45c0cf22e70c021dfbd"
server_key_iv: "209e4277d8bff1af3c407ebeccdbab31"
server_key_version: "encryptionkey@0"
I20260926 01:08:01.629797 16266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:01.648693 16266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:01.650892 16266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:01.651029 16266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:01.654838 16266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36245
I20260926 01:08:01.654886 16334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36245 every 8 connection(s)
I20260926 01:08:01.655253 16266 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:01.656172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 16266
I20260926 01:08:01.656529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39591
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36245
--webserver_port=42625
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:08:01.657150 16335 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:08:01.658030 16335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Bootstrap starting.
I20260926 01:08:01.666263 16335 log.cc:824] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:01.668229 16335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:08:01.668503 16335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Bootstrap complete.
I20260926 01:08:01.670739 16335 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:08:01.670989 16335 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bb3e8b1017410b9603e5c60e4e0379, State: Initialized, Role: FOLLOWER
I20260926 01:08:01.671139 16335 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:08:01.671202 16335 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:01.671248 16335 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:01.671296 16335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:08:01.672344 16335 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:08:01.672446 16335 leader_election.cc:302] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6bb3e8b1017410b9603e5c60e4e0379; no voters:
I20260926 01:08:01.672631 16335 leader_election.cc:288] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 01:08:01.672688 16339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:08:01.672816 16339 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 2 LEADER]: Becoming Leader. State: Replica: c6bb3e8b1017410b9603e5c60e4e0379, State: Running, Role: LEADER
I20260926 01:08:01.672919 16335 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:01.672986 16339 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:08:01.673380 16341 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } } }
I20260926 01:08:01.673594 16341 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:01.673323 16339 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6bb3e8b1017410b9603e5c60e4e0379. Latest consensus state: current_term: 2 leader_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } } }
I20260926 01:08:01.673673 16339 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:01.673858 16344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:01.674816 16344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:01.674966 16344 catalog_manager.cc:1259] Loaded cluster ID: b37ba87c7b6a4b84bc3de1ae5e622430
I20260926 01:08:01.675186 16344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:01.676342 16344 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:01.676524 16344 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Loaded TSK: 0
I20260926 01:08:01.689960 16344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Generated new TSK 1
I20260926 01:08:01.690093 16344 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:08:01.743157 16337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:01.743413 16337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:01.743450 16337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:01.744848 16337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:01.744903 16337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:01.744944 16337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:08:01.746660 16337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42625
--tserver_master_addrs=127.0.98.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:01.746922 16337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:01.747161 16337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:01.752609 16363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:01.752599 16365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:01.752599 16362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:01.753901 16337 server_base.cc:1034] running on GCE node
I20260926 01:08:01.754177 16337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:01.754545 16337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:01.755746 16337 hybrid_clock.cc:648] HybridClock initialized: now 1790384881755718 us; error 42 us; skew 500 ppm
Sep 26 01:08:01 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:01.759315 16337 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:08:01.759974 16337 webserver.cc:460] Webserver started at http://127.0.98.1:42625/ using document root <none> and password file <none>
I20260926 01:08:01.760334 16337 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:01.760438 16337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:01.762426 16337 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:08:01.763375 16372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:01.763581 16337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:08:01.763674 16337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "80a7d651276340fca1bfa4b4157c5703"
format_stamp: "Formatted at 2026-09-26 01:07:40 on dist-test-slave-l73v"
server_key: "09f24bf1a50c793ba9f5492888e691f9"
server_key_iv: "54ddc8bcb7f034060bce5faa6fdcca40"
server_key_version: "encryptionkey@0"
I20260926 01:08:01.764065 16337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:01.788642 16337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:01.791091 16337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:01.791221 16337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:01.791455 16337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:01.791920 16379 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:08:01.793388 16337 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:08:01.793434 16337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:08:01.793460 16337 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:08:01.794117 16337 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:08:01.794162 16337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:08:01.794236 16379 tablet_bootstrap.cc:492] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Bootstrap starting.
I20260926 01:08:01.801815 16337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39591
I20260926 01:08:01.802065 16486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39591 every 8 connection(s)
I20260926 01:08:01.802366 16337 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:08:01.805074 16379 log.cc:824] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:01.807762 16379 tablet_bootstrap.cc:492] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:08:01.808039 16379 tablet_bootstrap.cc:492] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Bootstrap complete.
I20260926 01:08:01.808848 16379 ts_tablet_manager.cc:1397] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260926 01:08:01.810215 16379 raft_consensus.cc:348] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:01.810534 16379 raft_consensus.cc:729] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a7d651276340fca1bfa4b4157c5703, State: Initialized, Role: FOLLOWER
I20260926 01:08:01.810705 16379 consensus_queue.cc:260] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:01.811008 16379 ts_tablet_manager.cc:1428] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 26 01:08:01 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:01.812261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 16337
I20260926 01:08:01.812582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44351
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36245
--webserver_port=37655
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:08:01.816535 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:01.803283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54625 (local address 127.0.98.62:36245)
0926 01:08:01.804419 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:01.804428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:01.804468 (+ 40us) server_negotiation.cc:407] Connection header received
0926 01:08:01.804604 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:01.804610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:01.804666 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:01.804805 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:01.805614 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.807017 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.808688 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.809289 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.809419 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:01.812180 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:01.812185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:01.812185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:01.812200 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:01.814327 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:01.814961 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:01.814965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:01.814965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:01.815075 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:01.815392 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:01.815394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:01.815394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:01.815733 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:01.815821 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:01.816187 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 01:08:01.816336 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077,"thread_start_us":94,"threads_started":1}
I20260926 01:08:01.816541 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:01.803147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:54625)
0926 01:08:01.803434 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:01.803478 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:01.804083 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:01.804881 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:01.804886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:01.805247 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:01.805466 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:01.805472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.807182 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.807184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:01.808564 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:01.808569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.809403 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.809404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:01.809557 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:01.810200 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:01.810210 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:01.812019 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:01.814454 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:01.814464 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:01.814467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:01.814816 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:01.815158 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:01.815159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:01.815159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:01.815316 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:01.815817 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:01.815824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:01.815999 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 01:08:01.816115 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 01:08:01.816317 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":89,"threads_started":1}
I20260926 01:08:01.817413 16487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36245
I20260926 01:08:01.817531 16487 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:01.817759 16487 heartbeater.cc:511] Master 127.0.98.62:36245 requested a full tablet report, sending...
I20260926 01:08:01.818774 16299 ts_manager.cc:194] Registered new tserver with Master: 80a7d651276340fca1bfa4b4157c5703 (127.0.98.1:39591)
I20260926 01:08:01.819931 16299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54625
I20260926 01:08:01.821079 16487 heartbeater.cc:503] Master 127.0.98.62:36245 was elected leader, sending a full tablet report...
Sep 26 01:08:01 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:08:01.828197 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:01.819503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39591 (local address 127.0.0.1:51902)
0926 01:08:01.819861 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:01.819899 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:01.820000 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:01.820301 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:01.820306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:01.820650 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:01.820707 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:01.820710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.821654 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.821656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:01.822318 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:01.822321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.822798 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.822802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:01.822881 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:01.823511 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:01.823517 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:01.825070 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:01.826998 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:01.827008 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:01.827011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:01.827233 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:01.827436 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:01.827437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:01.827437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:01.827504 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:01.827781 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:01.827790 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:01.827852 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 01:08:01.827925 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:08:01.828079 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":121,"threads_started":1}
I20260926 01:08:01.828197 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:01.819594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.0.98.1:39591)
0926 01:08:01.819790 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:01.819795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:01.820049 (+ 254us) server_negotiation.cc:407] Connection header received
0926 01:08:01.820184 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:01.820188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:01.820206 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:01.820281 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:01.820799 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.821586 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.822418 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.822725 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.822787 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:01.825172 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:01.825180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:01.825180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:01.825187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:01.826901 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:01.827313 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:01.827316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:01.827317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:01.827376 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:01.827557 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:01.827558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:01.827558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:01.827719 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:01.827793 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:01.827970 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 01:08:01.828101 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
I20260926 01:08:01.828632 16421 tablet_service.cc:1514] Processing DeleteTablet for tablet cee4ec99f225490297b75ffe62ea281e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:07:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51902
I20260926 01:08:01.829154 16496 tablet_replica.cc:330] stopping tablet replica
I20260926 01:08:01.829257 16496 raft_consensus.cc:2229] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:08:01.829331 16496 raft_consensus.cc:2258] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:01.829634 16496 ts_tablet_manager.cc:1905] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:01.831939 16496 ts_tablet_manager.cc:1918] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:01.832020 16496 log.cc:1196] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal/wals/cee4ec99f225490297b75ffe62ea281e
I20260926 01:08:01.832222 16496 ts_tablet_manager.cc:1939] T cee4ec99f225490297b75ffe62ea281e P 80a7d651276340fca1bfa4b4157c5703: Deleting consensus metadata
I20260926 01:08:01.832769 16283 catalog_manager.cc:4841] TS 80a7d651276340fca1bfa4b4157c5703 (127.0.98.1:39591): tablet cee4ec99f225490297b75ffe62ea281e (table test-table [id=ca172170244847969f1d1576ccfd4c46]) successfully deleted
W20260926 01:08:01.899159 16493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:01.899458 16493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:01.899492 16493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:01.901348 16493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:01.901427 16493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:01.901476 16493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:08:01.903631 16493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37655
--tserver_master_addrs=127.0.98.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:01.903898 16493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:01.904103 16493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:01.909425 16501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:01.909406 16504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:01.909406 16502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:01.909734 16493 server_base.cc:1034] running on GCE node
I20260926 01:08:01.909940 16493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:01.910181 16493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:01.911353 16493 hybrid_clock.cc:648] HybridClock initialized: now 1790384881911339 us; error 27 us; skew 500 ppm
Sep 26 01:08:01 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:01.914039 16493 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:08:01.914575 16493 webserver.cc:460] Webserver started at http://127.0.98.2:37655/ using document root <none> and password file <none>
I20260926 01:08:01.914934 16493 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:01.915027 16493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:01.916538 16493 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:08:01.917279 16511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:01.917549 16493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:01.917642 16493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "619c729ef81341359bf4996ceba10ffb"
format_stamp: "Formatted at 2026-09-26 01:07:41 on dist-test-slave-l73v"
server_key: "6317c357fea140e25b0fadc8bb72bdca"
server_key_iv: "a27c51b0421b614a9fa35d62ea1196fd"
server_key_version: "encryptionkey@0"
I20260926 01:08:01.918105 16493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:01.926936 16493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:01.929178 16493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:01.929312 16493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:01.929512 16493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:01.929782 16493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:01.929814 16493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:01.929836 16493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:01.929852 16493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:01.935734 16493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44351
I20260926 01:08:01.935802 16624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44351 every 8 connection(s)
I20260926 01:08:01.936148 16493 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:08:01.938884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 16493
I20260926 01:08:01.939152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32815
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36245
--webserver_port=41197
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 01:08:01 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:01.946842 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:01.936872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51865 (local address 127.0.98.62:36245)
0926 01:08:01.936927 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:01.936929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:01.937166 (+ 237us) server_negotiation.cc:407] Connection header received
0926 01:08:01.937691 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:01.937692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:01.937709 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:01.937762 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:01.938948 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.939599 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.940637 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.940878 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.940939 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:01.943649 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:01.943651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:01.943652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:01.943663 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:01.945148 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:01.945743 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:01.945744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:01.945744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:01.945814 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:01.946131 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:01.946132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:01.946132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:01.946243 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:01.946288 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:01.946681 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 01:08:01.946771 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:08:01.947113 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:01.936871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:51865)
0926 01:08:01.937038 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:01.937075 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:01.937579 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:01.937833 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:01.937837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:01.938494 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:01.938819 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:01.938827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.939710 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.939711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:01.940542 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:01.940544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:01.940956 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:01.940957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:01.941084 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:01.941647 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:01.941657 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:01.943479 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:01.945255 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:01.945266 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:01.945269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:01.945634 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:01.945886 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:01.945887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:01.945887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:01.946068 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:01.946299 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:01.946305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:01.946536 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 01:08:01.946663 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 01:08:01.946858 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260926 01:08:01.947672 16625 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36245
I20260926 01:08:01.947765 16625 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:01.947942 16625 heartbeater.cc:511] Master 127.0.98.62:36245 requested a full tablet report, sending...
I20260926 01:08:01.948325 16299 ts_manager.cc:194] Registered new tserver with Master: 619c729ef81341359bf4996ceba10ffb (127.0.98.2:44351)
I20260926 01:08:01.948709 16299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51865
W20260926 01:08:02.023228 16628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:02.023471 16628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:02.023505 16628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:02.024744 16628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:02.024798 16628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:02.024842 16628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:08:02.026476 16628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41197
--tserver_master_addrs=127.0.98.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:02.026722 16628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:02.026948 16628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:02.032178 16636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:02.032178 16633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:02.032222 16634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:02.032209 16628 server_base.cc:1034] running on GCE node
I20260926 01:08:02.032596 16628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:02.032864 16628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:02.034013 16628 hybrid_clock.cc:648] HybridClock initialized: now 1790384882034002 us; error 40 us; skew 500 ppm
Sep 26 01:08:02 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:02.036842 16628 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:08:02.037289 16628 webserver.cc:460] Webserver started at http://127.0.98.3:41197/ using document root <none> and password file <none>
I20260926 01:08:02.037499 16628 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:02.037557 16628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:02.039083 16628 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 01:08:02.039801 16643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:02.040023 16628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:02.040112 16628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "92c995d3125d4be2838836de812a6abc"
format_stamp: "Formatted at 2026-09-26 01:07:41 on dist-test-slave-l73v"
server_key: "a358b6070c0179c1edb76ede71068eb1"
server_key_iv: "ebcb02637b2da8cce98bc87823f72684"
server_key_version: "encryptionkey@0"
I20260926 01:08:02.040464 16628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:02.065892 16628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:02.068249 16628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:02.068368 16628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:02.068583 16628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:02.068863 16628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:02.068893 16628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:02.068914 16628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:02.068928 16628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:02.074627 16628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32815
I20260926 01:08:02.074682 16756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32815 every 8 connection(s)
I20260926 01:08:02.075013 16628 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:08:02 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:02.084311 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.075748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34749 (local address 127.0.98.62:36245)
0926 01:08:02.075837 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:02.075840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:02.076011 (+ 171us) server_negotiation.cc:407] Connection header received
0926 01:08:02.076534 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:02.076535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:02.076547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:02.076636 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:02.077591 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.078092 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.079105 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.079297 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.079354 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.081576 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:02.081578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:02.081578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:02.081586 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:02.082966 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:02.083375 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:02.083376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:02.083376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:02.083429 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:02.083662 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:02.083663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:02.083663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:02.083781 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:02.083837 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:02.084118 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 01:08:02.084214 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 01:08:02.084479 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.075694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:34749)
0926 01:08:02.075884 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:02.075929 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:02.076433 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:02.076708 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:02.076713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:02.077285 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:02.077481 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:02.077485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.078191 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.078192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:02.079012 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:02.079014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.079368 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.079368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:02.079491 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.079989 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:02.079997 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:02.081423 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:02.083056 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:02.083067 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:02.083070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:02.083304 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:02.083487 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:02.083488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:02.083488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:02.083604 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:02.083835 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:02.083842 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:02.083994 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 01:08:02.084086 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 01:08:02.084253 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260926 01:08:02.084599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 16628
I20260926 01:08:02.085035 16757 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36245
I20260926 01:08:02.085139 16757 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:02.085340 16757 heartbeater.cc:511] Master 127.0.98.62:36245 requested a full tablet report, sending...
I20260926 01:08:02.085763 16299 ts_manager.cc:194] Registered new tserver with Master: 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815)
I20260926 01:08:02.086268 16299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34749
I20260926 01:08:02.090936 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.084979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.0.98.62:36245)
0926 01:08:02.085070 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:02.085072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:02.085210 (+ 138us) server_negotiation.cc:407] Connection header received
0926 01:08:02.085290 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:02.085291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:02.085302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:02.085348 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:02.085960 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.086610 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.087355 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.087540 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.087625 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.088713 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:02.088714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:02.088715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:02.088721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:02.090021 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:02.090352 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:02.090353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:02.090353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:02.090426 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:02.090569 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:02.090570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:02.090570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:02.090666 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:02.090780 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:02.090811 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 01:08:02.090870 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 01:08:02.091667 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:08:02.094302 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.091969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.0.98.62:36245)
0926 01:08:02.092024 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:02.092026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:02.092084 (+ 58us) server_negotiation.cc:407] Connection header received
0926 01:08:02.092166 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:02.092166 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:02.092206 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:02.092261 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:08:02.092357 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.092824 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.093629 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.093827 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.093880 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.094042 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:08:02.094094 (+ 52us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 01:08:02.094237 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58872: token expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 01:08:02.094363 16490 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58872: token expired
W20260926 01:08:02.094542 16228 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:36245)
I20260926 01:08:02.100236 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.094804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.0.98.62:36245)
0926 01:08:02.094843 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:02.094845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:02.094853 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:08:02.094904 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:02.094904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:02.094913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:02.094957 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:02.095425 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.095886 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.096569 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.096750 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.096802 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.097866 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:02.097867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:02.097868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:02.097875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:02.099196 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:02.099539 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:02.099540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:02.099540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:02.099593 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:02.099757 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:02.099758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:02.099758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:02.099867 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:02.099952 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:02.100058 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 01:08:02.100138 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:08:02.101428 16299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:08:02.103087 16299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:08:02.105340 16421 tablet_service.cc:1467] Processing CreateTablet for tablet fb4f992a662d43bea2f66d005c255b20 (DEFAULT_TABLE table=test-table [id=663a8354087e40bf8340333116202f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:02.105612 16421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4f992a662d43bea2f66d005c255b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:02.107167 16379 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: Bootstrap starting.
I20260926 01:08:02.107470 16379 tablet_bootstrap.cc:654] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:02.108019 16379 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: No bootstrap required, opened a new log
I20260926 01:08:02.108072 16379 ts_tablet_manager.cc:1397] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:02.108191 16379 raft_consensus.cc:348] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:02.108273 16379 raft_consensus.cc:374] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:02.108330 16379 raft_consensus.cc:729] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a7d651276340fca1bfa4b4157c5703, State: Initialized, Role: FOLLOWER
I20260926 01:08:02.108378 16379 consensus_queue.cc:260] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:02.108474 16379 ts_tablet_manager.cc:1428] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 01:08:02.109338 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.104890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32815 (local address 127.0.0.1:43944)
0926 01:08:02.104998 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:02.105009 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:02.105060 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:02.105521 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:02.105522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:02.105524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:02.105568 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:02.105569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.106775 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.106775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:02.108141 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:02.108144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.109062 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.109063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:02.109130 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.109152 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 01:08:02.109208 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:08:02.109261 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":26,"thread_start_us":67,"threads_started":1}
I20260926 01:08:02.109453 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.104843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44351 (local address 127.0.0.1:34928)
0926 01:08:02.104890 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:02.104902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:02.104945 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:02.105409 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:02.105410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:02.105412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:02.105466 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:02.105468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.106712 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.106713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:02.108130 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:02.108133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.109170 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.109171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:02.109279 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.109283 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:02.109328 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:08:02.109380 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":128}
I20260926 01:08:02.109464 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.105086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.0.98.3:32815)
0926 01:08:02.105282 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:02.105286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:02.105296 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:08:02.105332 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:02.105334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:02.105348 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:02.105422 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:02.105642 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.106708 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.108232 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.108938 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.109022 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.109088 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:02.109338 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 01:08:02.109386 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":111,"threads_started":1}
I20260926 01:08:02.110116 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.104979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.0.98.2:44351)
0926 01:08:02.105252 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:02.105256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:02.105266 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:08:02.105305 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:02.105307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:02.105321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:02.105393 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:02.105640 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.106639 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.108229 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.108984 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.109056 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.109118 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:02.109969 (+ 851us) server_negotiation.cc:299] Negotiation successful
0926 01:08:02.110032 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260926 01:08:02.110483 16691 tablet_service.cc:1467] Processing CreateTablet for tablet fb4f992a662d43bea2f66d005c255b20 (DEFAULT_TABLE table=test-table [id=663a8354087e40bf8340333116202f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:02.110878 16691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4f992a662d43bea2f66d005c255b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:02.110889 16559 tablet_service.cc:1467] Processing CreateTablet for tablet fb4f992a662d43bea2f66d005c255b20 (DEFAULT_TABLE table=test-table [id=663a8354087e40bf8340333116202f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:02.111200 16559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4f992a662d43bea2f66d005c255b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:02.114485 16766 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Bootstrap starting.
I20260926 01:08:02.115103 16766 tablet_bootstrap.cc:654] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:02.115618 16766 log.cc:824] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:02.115932 16767 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Bootstrap starting.
I20260926 01:08:02.116436 16766 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: No bootstrap required, opened a new log
I20260926 01:08:02.116528 16767 tablet_bootstrap.cc:654] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:02.116534 16766 ts_tablet_manager.cc:1397] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:08:02.117138 16767 log.cc:824] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:02.117810 16767 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: No bootstrap required, opened a new log
I20260926 01:08:02.117864 16767 ts_tablet_manager.cc:1397] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:08:02.118144 16766 raft_consensus.cc:348] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:02.118309 16766 raft_consensus.cc:374] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:02.118340 16766 raft_consensus.cc:729] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619c729ef81341359bf4996ceba10ffb, State: Initialized, Role: FOLLOWER
I20260926 01:08:02.118484 16766 consensus_queue.cc:260] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:02.118746 16766 ts_tablet_manager.cc:1428] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:08:02.118800 16625 heartbeater.cc:503] Master 127.0.98.62:36245 was elected leader, sending a full tablet report...
I20260926 01:08:02.119477 16767 raft_consensus.cc:348] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:02.119614 16767 raft_consensus.cc:374] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:02.119635 16767 raft_consensus.cc:729] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c995d3125d4be2838836de812a6abc, State: Initialized, Role: FOLLOWER
I20260926 01:08:02.119782 16767 consensus_queue.cc:260] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:02.120039 16767 ts_tablet_manager.cc:1428] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:08:02.120131 16757 heartbeater.cc:503] Master 127.0.98.62:36245 was elected leader, sending a full tablet report...
I20260926 01:08:02.147971 16770 raft_consensus.cc:482] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:08:02.148105 16770 raft_consensus.cc:504] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:02.148492 16770 leader_election.cc:288] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815), 80a7d651276340fca1bfa4b4157c5703 (127.0.98.1:39591)
I20260926 01:08:02.151906 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.148796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60101 (local address 127.0.98.3:32815)
0926 01:08:02.148839 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:02.148841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:02.148964 (+ 123us) server_negotiation.cc:407] Connection header received
0926 01:08:02.149026 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:02.149027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:02.149038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:02.149138 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:02.149322 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.149787 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.150897 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.151531 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.151611 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.151624 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:02.151792 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 01:08:02.151838 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:08:02.151937 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.148742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59587 (local address 127.0.98.1:39591)
0926 01:08:02.148879 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:02.148880 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:02.148891 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:08:02.148921 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:02.148922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:02.148950 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:02.149016 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:02.149212 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.149806 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.150955 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.151604 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.151673 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.151769 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:02.151814 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 01:08:02.151868 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 01:08:02.151937 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.148677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39591 (local address 127.0.98.2:59587)
0926 01:08:02.148759 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:02.148771 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:02.148816 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:02.149035 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:02.149037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:02.149039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:02.149119 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:02.149121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.149887 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.149888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:02.150846 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:02.150849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.151674 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.151675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:02.151751 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.151754 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:02.151803 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:08:02.151847 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":50}
I20260926 01:08:02.151945 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.148687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32815 (local address 127.0.98.2:60101)
0926 01:08:02.148908 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:02.148915 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:02.148964 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:02.149164 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:02.149165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:02.149167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:02.149246 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:02.149248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.149877 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.149878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:02.150813 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:02.150816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.151637 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.151638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:02.151703 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.151708 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:08:02.151766 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:08:02.151813 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":39,"thread_start_us":101,"threads_started":1}
I20260926 01:08:02.152333 16711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4f992a662d43bea2f66d005c255b20" candidate_uuid: "619c729ef81341359bf4996ceba10ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c995d3125d4be2838836de812a6abc" is_pre_election: true
I20260926 01:08:02.152324 16440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4f992a662d43bea2f66d005c255b20" candidate_uuid: "619c729ef81341359bf4996ceba10ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a7d651276340fca1bfa4b4157c5703" is_pre_election: true
I20260926 01:08:02.152487 16711 raft_consensus.cc:2454] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619c729ef81341359bf4996ceba10ffb in term 0.
I20260926 01:08:02.152487 16440 raft_consensus.cc:2454] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619c729ef81341359bf4996ceba10ffb in term 0.
I20260926 01:08:02.152720 16512 leader_election.cc:302] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619c729ef81341359bf4996ceba10ffb, 80a7d651276340fca1bfa4b4157c5703; no voters:
I20260926 01:08:02.152860 16770 raft_consensus.cc:2781] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:08:02.152913 16770 raft_consensus.cc:482] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:08:02.152930 16770 raft_consensus.cc:3037] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:02.153658 16770 raft_consensus.cc:504] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:02.153788 16770 leader_election.cc:288] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [CANDIDATE]: Term 1 election: Requested vote from peers 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815), 80a7d651276340fca1bfa4b4157c5703 (127.0.98.1:39591)
I20260926 01:08:02.153956 16711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4f992a662d43bea2f66d005c255b20" candidate_uuid: "619c729ef81341359bf4996ceba10ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c995d3125d4be2838836de812a6abc"
I20260926 01:08:02.153966 16440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4f992a662d43bea2f66d005c255b20" candidate_uuid: "619c729ef81341359bf4996ceba10ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a7d651276340fca1bfa4b4157c5703"
I20260926 01:08:02.154057 16440 raft_consensus.cc:3037] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:02.154045 16711 raft_consensus.cc:3037] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:02.154945 16711 raft_consensus.cc:2454] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619c729ef81341359bf4996ceba10ffb in term 1.
I20260926 01:08:02.154945 16440 raft_consensus.cc:2454] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619c729ef81341359bf4996ceba10ffb in term 1.
I20260926 01:08:02.155143 16512 leader_election.cc:302] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619c729ef81341359bf4996ceba10ffb, 80a7d651276340fca1bfa4b4157c5703; no voters:
I20260926 01:08:02.155298 16770 raft_consensus.cc:2781] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:02.155440 16770 raft_consensus.cc:686] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 LEADER]: Becoming Leader. State: Replica: 619c729ef81341359bf4996ceba10ffb, State: Running, Role: LEADER
I20260926 01:08:02.155525 16770 consensus_queue.cc:237] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:02.156455 16299 catalog_manager.cc:5495] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb reported cstate change: term changed from 0 to 1, leader changed from <none> to 619c729ef81341359bf4996ceba10ffb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "619c729ef81341359bf4996ceba10ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } health_report { overall_health: UNKNOWN } } }
I20260926 01:08:02.167721 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.165372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.98.2:44351)
0926 01:08:02.165420 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:02.165422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:02.165430 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:08:02.165466 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:02.165467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:02.165487 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:02.165550 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:08:02.165657 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.166139 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.166963 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:02.167166 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:02.167231 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:02.167376 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:08:02.167536 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:08:02.167617 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:02.167633 (+ 16us) server_negotiation.cc:299] Negotiation successful
0926 01:08:02.167654 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:08:02.169595 16440 raft_consensus.cc:1261] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Refusing update from remote peer 619c729ef81341359bf4996ceba10ffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:02.169595 16711 raft_consensus.cc:1261] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 1 FOLLOWER]: Refusing update from remote peer 619c729ef81341359bf4996ceba10ffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:02.169962 16773 consensus_queue.cc:1035] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:02.170099 16770 consensus_queue.cc:1035] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:02.172410 16777 mvcc.cc:205] Tried to move back new op lower bound from 7333416477363929088 to 7333416477309763584. Current Snapshot: MvccSnapshot[applied={T|T < 7333416477363929088}]
I20260926 01:08:02.174803 16298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:02.174901 16298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:02.175649 16298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Sending DeleteTablet for 3 replicas of tablet fb4f992a662d43bea2f66d005c255b20
I20260926 01:08:02.175984 16421 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4f992a662d43bea2f66d005c255b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51902
I20260926 01:08:02.176051 16559 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4f992a662d43bea2f66d005c255b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34928
I20260926 01:08:02.176156 16691 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4f992a662d43bea2f66d005c255b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43944
I20260926 01:08:02.176332 16496 tablet_replica.cc:330] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: stopping tablet replica
I20260926 01:08:02.176355 16787 tablet_replica.cc:330] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: stopping tablet replica
I20260926 01:08:02.176399 16496 raft_consensus.cc:2229] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:08:02.176436 16787 raft_consensus.cc:2229] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:08:02.176465 16496 raft_consensus.cc:2258] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:02.176600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 16337
I20260926 01:08:02.176644 16787 raft_consensus.cc:2258] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:02.177657 16787 ts_tablet_manager.cc:1905] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:02.179522 16787 ts_tablet_manager.cc:1918] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:02.179584 16787 log.cc:1196] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/fb4f992a662d43bea2f66d005c255b20
I20260926 01:08:02.179781 16787 ts_tablet_manager.cc:1939] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Deleting consensus metadata
I20260926 01:08:02.180186 16284 catalog_manager.cc:4841] TS 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815): tablet fb4f992a662d43bea2f66d005c255b20 (table test-table [id=663a8354087e40bf8340333116202f87]) successfully deleted
I20260926 01:08:02.182708 16786 tablet_replica.cc:330] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: stopping tablet replica
I20260926 01:08:02.182869 16786 raft_consensus.cc:2229] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 LEADER]: Raft consensus shutting down.
W20260926 01:08:02.182905 16283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39591 (error 108)
I20260926 01:08:02.182987 16786 raft_consensus.cc:2258] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:02.183317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 16493
I20260926 01:08:02.183712 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.183602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39591 (local address 127.0.0.1:51910)
0926 01:08:02.183650 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:02.183678 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 01:08:02.183894 16283 catalog_manager.cc:4584] TS 80a7d651276340fca1bfa4b4157c5703 (127.0.98.1:39591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb4f992a662d43bea2f66d005c255b20: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39591: connect: Connection refused (error 111)
I20260926 01:08:02.189246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 16628
I20260926 01:08:02.189364 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:02.189244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44351 (local address 127.0.0.1:34950)
0926 01:08:02.189294 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:02.189323 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:08:02.193830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 16266
I20260926 01:08:22.199301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36245
--webserver_interface=127.0.98.62
--webserver_port=46585
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:22.280393 16789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:22.280660 16789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:22.280689 16789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:22.281937 16789 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 01:08:22.281975 16789 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 01:08:22.281986 16789 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 01:08:22.281997 16789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:22.282008 16789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:22.282020 16789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:22.282032 16789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:22.282042 16789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:22.283722 16789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:22.283943 16789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:22.284138 16789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:22.289060 16797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:22.289021 16794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:22.289039 16795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:22.289304 16789 server_base.cc:1034] running on GCE node
I20260926 01:08:22.289567 16789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:22.289882 16789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:22.291047 16789 hybrid_clock.cc:648] HybridClock initialized: now 1790384902291024 us; error 46 us; skew 500 ppm
Sep 26 01:08:22 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:22.298753 16789 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:08:22.299252 16789 webserver.cc:460] Webserver started at http://127.0.98.62:46585/ using document root <none> and password file <none>
I20260926 01:08:22.299467 16789 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:22.299516 16789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:22.300959 16789 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:22.301716 16804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:22.301884 16789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:22.301963 16789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "c6bb3e8b1017410b9603e5c60e4e0379"
format_stamp: "Formatted at 2026-09-26 01:07:40 on dist-test-slave-l73v"
server_key: "e347a2b1bd63e45c0cf22e70c021dfbd"
server_key_iv: "209e4277d8bff1af3c407ebeccdbab31"
server_key_version: "encryptionkey@0"
I20260926 01:08:22.302316 16789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:22.330688 16789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:22.332923 16789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:22.333039 16789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:22.337260 16789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36245
I20260926 01:08:22.337306 16856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36245 every 8 connection(s)
I20260926 01:08:22.337675 16789 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:22.339316 16857 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:08:22.339970 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Bootstrap starting.
I20260926 01:08:22.344523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 16789
I20260926 01:08:22.344795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39591
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36245
--webserver_port=42625
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:08:22.348730 16857 log.cc:824] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:22.351042 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:08:22.351329 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Bootstrap complete.
I20260926 01:08:22.353443 16857 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:08:22.353683 16857 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bb3e8b1017410b9603e5c60e4e0379, State: Initialized, Role: FOLLOWER
I20260926 01:08:22.353847 16857 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:08:22.353927 16857 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:22.353981 16857 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:22.354029 16857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:08:22.355168 16857 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:08:22.355276 16857 leader_election.cc:302] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6bb3e8b1017410b9603e5c60e4e0379; no voters:
I20260926 01:08:22.355455 16857 leader_election.cc:288] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 01:08:22.355500 16861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 01:08:22.355619 16861 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [term 3 LEADER]: Becoming Leader. State: Replica: c6bb3e8b1017410b9603e5c60e4e0379, State: Running, Role: LEADER
I20260926 01:08:22.355700 16861 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } }
I20260926 01:08:22.355741 16857 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:22.355964 16862 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } } }
I20260926 01:08:22.356065 16862 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:22.356292 16863 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6bb3e8b1017410b9603e5c60e4e0379. Latest consensus state: current_term: 3 leader_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bb3e8b1017410b9603e5c60e4e0379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36245 } } }
I20260926 01:08:22.356374 16863 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:22.356511 16866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:22.357692 16866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:22.357798 16866 catalog_manager.cc:1259] Loaded cluster ID: b37ba87c7b6a4b84bc3de1ae5e622430
I20260926 01:08:22.357820 16866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:22.359067 16866 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:22.359261 16866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Loaded TSK: 0
I20260926 01:08:22.359306 16866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6bb3e8b1017410b9603e5c60e4e0379: Loaded TSK: 1
F20260926 01:08:22.359526 16866 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efcc17f19ad google::LogMessage::Fail() at ??:0
@ 0x7efcc17f488a google::LogMessage::SendToLog() at ??:0
@ 0x7efcc17f14de google::LogMessage::Flush() at ??:0
@ 0x7efcc17f2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efcc3f5e104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7efcc3f5bd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7efcc974b1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7efcc974b48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7efcc974b594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7efcc9711c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7efcc971bad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efcc36bf0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efcc36b4954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7efcc45186db start_thread at ??:0
@ 0x7efcc02ef61f clone at ??:0
W20260926 01:08:22.462072 16859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:22.462432 16859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:22.462481 16859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:22.464366 16859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:22.464444 16859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:22.464491 16859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:08:22.466953 16859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42625
--tserver_master_addrs=127.0.98.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:22.467255 16859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:22.467540 16859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:22.473109 16901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:22.473114 16898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:22.473131 16899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:22.473335 16859 server_base.cc:1034] running on GCE node
I20260926 01:08:22.473728 16859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:22.473999 16859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:22.475162 16859 hybrid_clock.cc:648] HybridClock initialized: now 1790384902475147 us; error 32 us; skew 500 ppm
Sep 26 01:08:22 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:22.478158 16859 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:08:22.478816 16859 webserver.cc:460] Webserver started at http://127.0.98.1:42625/ using document root <none> and password file <none>
I20260926 01:08:22.479092 16859 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:22.479153 16859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:22.480911 16859 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:08:22.481758 16908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:22.481951 16859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:22.482055 16859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "80a7d651276340fca1bfa4b4157c5703"
format_stamp: "Formatted at 2026-09-26 01:07:40 on dist-test-slave-l73v"
server_key: "09f24bf1a50c793ba9f5492888e691f9"
server_key_iv: "54ddc8bcb7f034060bce5faa6fdcca40"
server_key_version: "encryptionkey@0"
I20260926 01:08:22.482546 16859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:22.500483 16859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:22.502839 16859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:22.502991 16859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:22.503223 16859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:22.503680 16915 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:08:22.505759 16859 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:08:22.505838 16859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:08:22.505879 16859 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:08:22.506675 16859 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:08:22.506711 16859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:22.506803 16915 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: Bootstrap starting.
I20260926 01:08:22.513576 16859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39591
I20260926 01:08:22.513659 17022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39591 every 8 connection(s)
I20260926 01:08:22.514002 16859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:08:22.518476 16915 log.cc:824] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:22.520502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 16859
I20260926 01:08:22.520756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44351
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36245
--webserver_port=37655
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:08:22.521410 16915 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:08:22.521713 16915 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: Bootstrap complete.
I20260926 01:08:22.522891 16915 ts_tablet_manager.cc:1397] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.015s
I20260926 01:08:22.524456 16915 raft_consensus.cc:348] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:22.524818 16915 raft_consensus.cc:729] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a7d651276340fca1bfa4b4157c5703, State: Initialized, Role: FOLLOWER
I20260926 01:08:22.524981 16915 consensus_queue.cc:260] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:22.525300 16915 ts_tablet_manager.cc:1428] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260926 01:08:22.585449 17025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:08:22.514715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:40307)
0926 01:08:22.514972 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.515012 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:22.515479 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:22.585260 (+ 69781us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
W20260926 01:08:22.585681 17023 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36245 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36245: Client connection negotiation failed: client connection to 127.0.98.62:36245: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:08:22.585987 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:22.585880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:52899)
0926 01:08:22.585927 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.585968 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:08:22.586457 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:22.586292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:35857)
0926 01:08:22.586373 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.586427 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 01:08:22.586601 17023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 01:08:22.605358 17027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:22.605566 17027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:22.605590 17027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:22.606861 17027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:22.606907 17027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:22.606935 17027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:08:22.608433 17027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37655
--tserver_master_addrs=127.0.98.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:22.608673 17027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:22.608906 17027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:22.614168 17036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:22.614172 17034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:22.614172 17033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:22.614815 17027 server_base.cc:1034] running on GCE node
I20260926 01:08:22.615046 17027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:22.615305 17027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:22.616514 17027 hybrid_clock.cc:648] HybridClock initialized: now 1790384902616498 us; error 29 us; skew 500 ppm
Sep 26 01:08:22 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:22.619056 17027 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:08:22.619562 17027 webserver.cc:460] Webserver started at http://127.0.98.2:37655/ using document root <none> and password file <none>
I20260926 01:08:22.619784 17027 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:22.619835 17027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:22.621244 17027 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:22.621997 17043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:22.622172 17027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:22.622255 17027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "619c729ef81341359bf4996ceba10ffb"
format_stamp: "Formatted at 2026-09-26 01:07:41 on dist-test-slave-l73v"
server_key: "6317c357fea140e25b0fadc8bb72bdca"
server_key_iv: "a27c51b0421b614a9fa35d62ea1196fd"
server_key_version: "encryptionkey@0"
I20260926 01:08:22.622697 17027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:22.631646 17027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:22.633834 17027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:22.633981 17027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:22.634208 17027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:22.634673 17050 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 01:08:22.636268 17027 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 01:08:22.636330 17027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 01:08:22.636368 17027 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 01:08:22.637090 17027 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 01:08:22.637125 17027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 01:08:22.637216 17050 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Bootstrap starting.
I20260926 01:08:22.642978 17027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44351
I20260926 01:08:22.643203 17157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44351 every 8 connection(s)
I20260926 01:08:22.643435 17027 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:08:22.644706 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:22.644149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:46259)
0926 01:08:22.644340 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.644462 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
W20260926 01:08:22.644975 17158 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36245 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36245: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
I20260926 01:08:22.645308 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:22.645189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:40335)
0926 01:08:22.645247 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.645281 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:08:22.645612 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:22.645525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:52049)
0926 01:08:22.645566 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.645592 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:08:22.645763 17158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:08:22.646020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 17027
I20260926 01:08:22.646261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32815
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36245
--webserver_port=41197
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:08:22.647518 17050 log.cc:824] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:22.649729 17050 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:08:22.649943 17050 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Bootstrap complete.
I20260926 01:08:22.650840 17050 ts_tablet_manager.cc:1397] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.002s
I20260926 01:08:22.652607 17050 raft_consensus.cc:348] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:22.653000 17050 raft_consensus.cc:729] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619c729ef81341359bf4996ceba10ffb, State: Initialized, Role: FOLLOWER
I20260926 01:08:22.653163 17050 consensus_queue.cc:260] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:22.653453 17050 ts_tablet_manager.cc:1428] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 01:08:22.653779 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:22.653664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:51875)
0926 01:08:22.653713 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.653747 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:08:22.654119 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:22.654044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:34681)
0926 01:08:22.654082 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.654103 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 01:08:22.731146 17161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:22.731426 17161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:22.731456 17161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:22.732697 17161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:22.732743 17161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:22.732770 17161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:08:22.734283 17161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41197
--tserver_master_addrs=127.0.98.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:22.734529 17161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:22.734730 17161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:22.740212 17169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:22.740325 17161 server_base.cc:1034] running on GCE node
W20260926 01:08:22.740316 17171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:22.740212 17168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:22.740609 17161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:22.740880 17161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:22.742056 17161 hybrid_clock.cc:648] HybridClock initialized: now 1790384902742037 us; error 30 us; skew 500 ppm
Sep 26 01:08:22 dist-test-slave-l73v krb5kdc[15720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:22.744858 17161 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:08:22.745368 17161 webserver.cc:460] Webserver started at http://127.0.98.3:41197/ using document root <none> and password file <none>
I20260926 01:08:22.745591 17161 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:22.745641 17161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:22.747164 17161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:22.747829 17178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:22.748078 17161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:08:22.748178 17161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "92c995d3125d4be2838836de812a6abc"
format_stamp: "Formatted at 2026-09-26 01:07:41 on dist-test-slave-l73v"
server_key: "a358b6070c0179c1edb76ede71068eb1"
server_key_iv: "ebcb02637b2da8cce98bc87823f72684"
server_key_version: "encryptionkey@0"
I20260926 01:08:22.748543 17161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:22.762310 17161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:22.764662 17161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:22.764812 17161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:22.765046 17161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:22.765359 17161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:22.765393 17161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:22.765424 17161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:22.765439 17161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:22.771723 17161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32815
I20260926 01:08:22.771771 17291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32815 every 8 connection(s)
I20260926 01:08:22.772116 17161 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:08:22.773176 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:22.772809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:42547)
0926 01:08:22.772960 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.773033 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
W20260926 01:08:22.773640 17292 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36245 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36245: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
I20260926 01:08:22.773979 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:22.773864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:36703)
0926 01:08:22.773917 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.773952 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 01:08:22.773965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 17161
I20260926 01:08:22.774351 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:22.774238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:52915)
0926 01:08:22.774312 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:22.774335 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 01:08:22.774570 17292 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 01:08:23.587218 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:23.586974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:40523)
0926 01:08:23.587103 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:23.587153 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260926 01:08:23.654821 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:23.654571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:52547)
0926 01:08:23.654710 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:23.654758 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260926 01:08:23.775171 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:23.774930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:53645)
0926 01:08:23.775064 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:23.775106 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260926 01:08:23.877048 17299 raft_consensus.cc:482] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:08:23.877257 17299 raft_consensus.cc:504] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:23.877790 17299 leader_election.cc:288] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815), 80a7d651276340fca1bfa4b4157c5703 (127.0.98.1:39591)
Sep 26 01:08:23 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 01:08:23 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 01:08:23.888604 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:23.878077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50653 (local address 127.0.98.3:32815)
0926 01:08:23.878412 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:23.878419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:23.878452 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:08:23.879016 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:23.879020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:23.879072 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:23.879215 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:23.879619 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:23.880788 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:23.881854 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:23.882264 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:23.882370 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:23.884901 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:23.884906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:23.884907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:23.884918 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:23.886760 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:23.887290 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:23.887295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:23.887295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:23.887423 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:23.887676 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:23.887677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:23.887678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:23.887971 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:23.888064 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:23.888349 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 01:08:23.888478 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":47,"threads_started":1}
I20260926 01:08:23.888592 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:23.878030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39591 (local address 127.0.98.2:44303)
0926 01:08:23.878157 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:23.878173 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:23.878373 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:23.878677 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:23.878681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:23.879060 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:23.879332 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:23.879338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:23.880908 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:23.880909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:23.881758 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:23.881761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:23.882358 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:23.882359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:23.882501 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:23.883093 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:23.883097 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:23.885059 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:23.886911 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:23.886913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:23.886913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:23.887208 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:23.887487 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:23.887488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:23.887488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:23.887616 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:23.888043 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:23.888050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:23.888182 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 01:08:23.888274 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 01:08:23.888486 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":32,"thread_start_us":67,"threads_started":1}
I20260926 01:08:23.888859 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:23.877968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32815 (local address 127.0.98.2:50653)
0926 01:08:23.878028 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:23.878174 (+ 146us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:23.878354 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:23.879219 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:23.879220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:23.879412 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:23.879478 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:23.879480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:23.880894 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:23.880895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:23.881760 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:23.881762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:23.882353 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:23.882354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:23.882501 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:23.883065 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:23.883074 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:23.884696 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:23.886856 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:23.886867 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:23.886870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:23.887208 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:23.887487 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:23.887488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:23.887488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:23.887618 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:23.888090 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:23.888091 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:23.888181 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 01:08:23.888265 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 01:08:23.888559 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":199}
I20260926 01:08:23.888633 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:23.878090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44303 (local address 127.0.98.1:39591)
0926 01:08:23.878246 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:23.878251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:23.878265 (+ 14us) server_negotiation.cc:407] Connection header received
0926 01:08:23.878534 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:23.878537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:23.878557 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:23.878623 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:23.879464 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:23.880787 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:23.881856 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:23.882270 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:23.882370 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:23.885166 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:23.885170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:23.885170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:23.885181 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:23.886824 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:23.887311 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:23.887315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:23.887315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:23.887429 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:23.887673 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:23.887673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:23.887674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:23.887974 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:23.888051 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:23.888378 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 01:08:23.888529 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":50,"threads_started":1}
I20260926 01:08:23.889283 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4f992a662d43bea2f66d005c255b20" candidate_uuid: "619c729ef81341359bf4996ceba10ffb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "80a7d651276340fca1bfa4b4157c5703" is_pre_election: true
I20260926 01:08:23.889436 16977 raft_consensus.cc:2454] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619c729ef81341359bf4996ceba10ffb in term 1.
I20260926 01:08:23.889731 17044 leader_election.cc:302] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619c729ef81341359bf4996ceba10ffb, 80a7d651276340fca1bfa4b4157c5703; no voters:
I20260926 01:08:23.889897 17299 raft_consensus.cc:2781] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:08:23.889966 17299 raft_consensus.cc:482] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:08:23.889991 17299 raft_consensus.cc:3037] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 1 FOLLOWER]: Advancing to term 2
W20260926 01:08:23.890033 17045 leader_election.cc:341] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815): Not found: Tablet not found: fb4f992a662d43bea2f66d005c255b20
I20260926 01:08:23.891355 17299 raft_consensus.cc:504] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:23.891496 17299 leader_election.cc:288] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [CANDIDATE]: Term 2 election: Requested vote from peers 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815), 80a7d651276340fca1bfa4b4157c5703 (127.0.98.1:39591)
I20260926 01:08:23.891701 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4f992a662d43bea2f66d005c255b20" candidate_uuid: "619c729ef81341359bf4996ceba10ffb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "80a7d651276340fca1bfa4b4157c5703"
I20260926 01:08:23.891803 16977 raft_consensus.cc:3037] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 1 FOLLOWER]: Advancing to term 2
W20260926 01:08:23.891886 17045 leader_election.cc:341] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815): Not found: Tablet not found: fb4f992a662d43bea2f66d005c255b20
I20260926 01:08:23.892987 16977 raft_consensus.cc:2454] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619c729ef81341359bf4996ceba10ffb in term 2.
I20260926 01:08:23.893254 17044 leader_election.cc:302] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 619c729ef81341359bf4996ceba10ffb, 80a7d651276340fca1bfa4b4157c5703; no voters: 92c995d3125d4be2838836de812a6abc
I20260926 01:08:23.893404 17299 raft_consensus.cc:2781] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:08:23.893535 17299 raft_consensus.cc:686] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [term 2 LEADER]: Becoming Leader. State: Replica: 619c729ef81341359bf4996ceba10ffb, State: Running, Role: LEADER
I20260926 01:08:23.893630 17299 consensus_queue.cc:237] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:23.893875 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:23.893761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:55693)
0926 01:08:23.893802 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:23.893842 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 01:08:24.313259 17045 consensus_peers.cc:487] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb -> Peer 92c995d3125d4be2838836de812a6abc (127.0.98.3:32815): Couldn't send request to peer 92c995d3125d4be2838836de812a6abc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb4f992a662d43bea2f66d005c255b20. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:08:24.319239 16977 raft_consensus.cc:1261] T fb4f992a662d43bea2f66d005c255b20 P 80a7d651276340fca1bfa4b4157c5703 [term 2 FOLLOWER]: Refusing update from remote peer 619c729ef81341359bf4996ceba10ffb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 01:08:24.319667 17299 consensus_queue.cc:1035] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 01:08:24.320662 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:24.320480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:43531)
0926 01:08:24.320568 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:24.320617 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":44,"threads_started":1}
I20260926 01:08:24.321393 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:24.321289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:58007)
0926 01:08:24.321333 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:24.321360 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:08:24.321498 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:24.321410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:32883)
0926 01:08:24.321445 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:24.321471 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 01:08:24.775911 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:24.775611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:57793)
0926 01:08:24.775768 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:24.775822 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260926 01:08:24.892864 17311 ts_tablet_manager.cc:927] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Initiating tablet copy from peer 619c729ef81341359bf4996ceba10ffb (127.0.98.2:44351)
I20260926 01:08:24.893162 17311 tablet_copy_client.cc:316] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:44351
Sep 26 01:08:24 dist-test-slave-l73v krb5kdc[15720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 01:08:24.901687 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:24.893589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45039 (local address 127.0.98.2:44351)
0926 01:08:24.893696 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:24.893701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:24.893711 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:08:24.893737 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:24.893738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:24.893751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:24.893827 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:24.894326 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:24.895088 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:24.895949 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:24.896287 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:24.896360 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:24.898646 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:24.898656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:24.898657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:24.898664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:24.900345 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:24.900804 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:24.900808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:24.900808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:24.900877 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:24.901101 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:24.901101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:24.901102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:24.901258 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:24.901362 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:24.901533 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 01:08:24.901612 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":43,"threads_started":1}
I20260926 01:08:24.901717 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:24.893479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44351 (local address 127.0.98.3:45039)
0926 01:08:24.893548 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:24.893565 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:24.893623 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:24.893869 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:24.893877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:24.894168 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:24.894253 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:24.894255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:24.895184 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:24.895185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:24.895847 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:24.895850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:24.896375 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:24.896377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:24.896479 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:24.897054 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:24.897059 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:24.898531 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:24.900449 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:24.900457 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:24.900459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:24.900693 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:24.900968 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:24.900969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:24.900969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:24.901041 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:24.901369 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:24.901381 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:24.901437 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 01:08:24.901507 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:08:24.901619 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 01:08:24.902221 17132 tablet_copy_service.cc:140] P 619c729ef81341359bf4996ceba10ffb: Received BeginTabletCopySession request for tablet fb4f992a662d43bea2f66d005c255b20 from peer 92c995d3125d4be2838836de812a6abc ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45039)
I20260926 01:08:24.902297 17132 tablet_copy_service.cc:161] P 619c729ef81341359bf4996ceba10ffb: Beginning new tablet copy session on tablet fb4f992a662d43bea2f66d005c255b20 from peer 92c995d3125d4be2838836de812a6abc at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45039: session id = 92c995d3125d4be2838836de812a6abc-fb4f992a662d43bea2f66d005c255b20
I20260926 01:08:24.902796 17132 tablet_copy_source_session.cc:205] T fb4f992a662d43bea2f66d005c255b20 P 619c729ef81341359bf4996ceba10ffb: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 01:08:24.903887 17311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4f992a662d43bea2f66d005c255b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:24.906652 17311 tablet_copy_client.cc:791] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: tablet copy: Starting download of 0 data blocks...
I20260926 01:08:24.906816 17311 tablet_copy_client.cc:655] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: tablet copy: Starting download of 1 WAL segments...
I20260926 01:08:24.907802 17311 tablet_copy_client.cc:531] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 01:08:24.908532 17311 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Bootstrap starting.
I20260926 01:08:24.910022 17311 log.cc:824] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:24.912729 17311 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:08:24.912964 17311 tablet_bootstrap.cc:492] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Bootstrap complete.
I20260926 01:08:24.913092 17311 ts_tablet_manager.cc:1397] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260926 01:08:24.914111 17311 raft_consensus.cc:348] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:24.914295 17311 raft_consensus.cc:729] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c995d3125d4be2838836de812a6abc, State: Initialized, Role: FOLLOWER
I20260926 01:08:24.914471 17311 consensus_queue.cc:260] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c729ef81341359bf4996ceba10ffb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44351 } } peers { permanent_uuid: "92c995d3125d4be2838836de812a6abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32815 } } peers { permanent_uuid: "80a7d651276340fca1bfa4b4157c5703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39591 } }
I20260926 01:08:24.914764 17311 ts_tablet_manager.cc:1428] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260926 01:08:24.915160 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:24.915036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:55845)
0926 01:08:24.915088 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:24.915127 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 01:08:24.915190 17132 tablet_copy_service.cc:342] P 619c729ef81341359bf4996ceba10ffb: Request end of tablet copy session 92c995d3125d4be2838836de812a6abc-fb4f992a662d43bea2f66d005c255b20 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45039
I20260926 01:08:24.915272 17132 tablet_copy_service.cc:433] P 619c729ef81341359bf4996ceba10ffb: ending tablet copy session 92c995d3125d4be2838836de812a6abc-fb4f992a662d43bea2f66d005c255b20 on tablet fb4f992a662d43bea2f66d005c255b20 with peer 92c995d3125d4be2838836de812a6abc
I20260926 01:08:25.322125 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:25.321893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:54093)
0926 01:08:25.322020 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:25.322067 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260926 01:08:25.322125 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:25.321893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:35389)
0926 01:08:25.322046 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:25.322085 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260926 01:08:25.409430 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:25.409296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:35945)
0926 01:08:25.409346 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:25.409382 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:08:25.409534 17246 raft_consensus.cc:1203] T fb4f992a662d43bea2f66d005c255b20 P 92c995d3125d4be2838836de812a6abc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 01:08:25.409991 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:25.409877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:38345)
0926 01:08:25.409927 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:25.409961 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:08:26.322921 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:26.322646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:58969)
0926 01:08:26.322827 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:26.322873 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":94,"threads_started":1}
I20260926 01:08:26.410403 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:26.410069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:44139)
0926 01:08:26.410272 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:26.410325 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20260926 01:08:26.410669 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:26.410468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:56037)
0926 01:08:26.410587 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:26.410624 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20260926 01:08:27.323781 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:27.323463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:43407)
0926 01:08:27.323684 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:27.323734 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260926 01:08:27.411334 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:27.411013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:46279)
0926 01:08:27.411247 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:27.411288 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":161,"threads_started":1}
I20260926 01:08:27.411334 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:27.411089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:38151)
0926 01:08:27.411252 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:27.411292 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260926 01:08:28.324714 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:28.324421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:48999)
0926 01:08:28.324620 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:28.324668 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260926 01:08:28.412209 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:28.411964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:60013)
0926 01:08:28.412124 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:28.412161 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260926 01:08:28.412232 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:28.411964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:34681)
0926 01:08:28.412141 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:28.412190 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20260926 01:08:28.412621 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:28.412528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:60583)
0926 01:08:28.412571 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:28.412598 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 01:08:29.325646 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:29.325395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:50421)
0926 01:08:29.325550 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:29.325594 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260926 01:08:29.413148 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:29.412808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:35719)
0926 01:08:29.413007 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:29.413079 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":66,"threads_started":1}
I20260926 01:08:29.413484 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:29.413193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:41151)
0926 01:08:29.413389 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:29.413433 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260926 01:08:30.326524 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:30.326234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:60285)
0926 01:08:30.326427 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:30.326475 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":64,"threads_started":1}
I20260926 01:08:30.414172 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:30.413945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:57629)
0926 01:08:30.414088 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:30.414122 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260926 01:08:30.414374 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:30.413964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:55259)
0926 01:08:30.414278 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:30.414326 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":58,"threads_started":1}
I20260926 01:08:31.327244 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:31.327032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:42319)
0926 01:08:31.327150 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:31.327197 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260926 01:08:31.414916 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:31.414693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:38175)
0926 01:08:31.414830 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:31.414868 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260926 01:08:31.415108 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:31.414884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:49739)
0926 01:08:31.415012 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:31.415059 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260926 01:08:32.328023 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:32.327761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:50621)
0926 01:08:32.327927 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:32.327976 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260926 01:08:32.415529 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:32.415335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:39635)
0926 01:08:32.415449 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:32.415482 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260926 01:08:32.415692 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:32.415494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:40491)
0926 01:08:32.415614 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:32.415649 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":50,"threads_started":1}
I20260926 01:08:33.328878 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:33.328617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:55225)
0926 01:08:33.328780 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:33.328831 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260926 01:08:33.416352 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:33.416125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:59025)
0926 01:08:33.416268 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:33.416300 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260926 01:08:33.416355 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:33.416129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:52647)
0926 01:08:33.416263 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:33.416304 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260926 01:08:34.329595 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:34.329366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:37409)
0926 01:08:34.329510 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:34.329550 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
I20260926 01:08:34.417147 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:34.416888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:60665)
0926 01:08:34.417047 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:34.417097 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260926 01:08:34.417140 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:34.416888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:57033)
0926 01:08:34.417059 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:34.417092 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
I20260926 01:08:35.330473 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:35.330164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:39647)
0926 01:08:35.330343 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:35.330418 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
I20260926 01:08:35.417918 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:35.417695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:51601)
0926 01:08:35.417831 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:35.417870 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260926 01:08:35.417954 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:35.417713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:34697)
0926 01:08:35.417869 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:35.417905 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260926 01:08:36.331274 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:36.331021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:43295)
0926 01:08:36.331161 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:36.331208 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260926 01:08:36.418697 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:36.418491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:51081)
0926 01:08:36.418608 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:36.418645 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260926 01:08:36.418694 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:36.418457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:55827)
0926 01:08:36.418602 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:36.418645 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260926 01:08:37.332106 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:37.331846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.1:44295)
0926 01:08:37.332014 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:37.332061 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260926 01:08:37.419476 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:37.419194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.2:43843)
0926 01:08:37.419388 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:37.419427 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
I20260926 01:08:37.419483 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:37.419194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36245 (local address 127.0.98.3:53883)
0926 01:08:37.419390 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:37.419439 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260926 01:08:37.775269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 16859
I20260926 01:08:37.780133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 17027
I20260926 01:08:37.784667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 17161
2026-09-26T01:08:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58293 ms)
[----------] 30 tests from SecurityITest (237509 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:37 dist-test-slave-l73v krb5kdc[17359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:37 dist-test-slave-l73v krb5kdc[17359](info): set up 2 sockets
Sep 26 01:08:37 dist-test-slave-l73v krb5kdc[17359](info): commencing operation
krb5kdc: starting...
W20260926 01:08:38.745980 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:38 dist-test-slave-l73v krb5kdc[17359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:08:38.866130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39273
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:38.949059 17372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:38.949295 17372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:38.949321 17372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:38.950549 17372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:38.950588 17372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:38.950601 17372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:38.950613 17372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:38.950623 17372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:38.952136 17372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:38.952353 17372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:38.952551 17372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:38.957908 17377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:38.957908 17380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:38.958012 17378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:38.958066 17372 server_base.cc:1034] running on GCE node
I20260926 01:08:38.958446 17372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:38.958807 17372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:38.959960 17372 hybrid_clock.cc:648] HybridClock initialized: now 1790384918959950 us; error 31 us; skew 500 ppm
Sep 26 01:08:38 dist-test-slave-l73v krb5kdc[17359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:38.962502 17372 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:08:38.962999 17372 webserver.cc:460] Webserver started at http://127.0.98.62:45771/ using document root <none> and password file <none>
I20260926 01:08:38.963203 17372 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:38.963248 17372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:38.963372 17372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:38.964409 17372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268"
format_stamp: "Formatted at 2026-09-26 01:08:38 on dist-test-slave-l73v"
server_key: "cc0d7da4c0e106df893668022febb66d"
server_key_iv: "8225ef005552ffa91752a3f9716ee8d5"
server_key_version: "encryptionkey@0"
I20260926 01:08:38.964754 17372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268"
format_stamp: "Formatted at 2026-09-26 01:08:38 on dist-test-slave-l73v"
server_key: "cc0d7da4c0e106df893668022febb66d"
server_key_iv: "8225ef005552ffa91752a3f9716ee8d5"
server_key_version: "encryptionkey@0"
I20260926 01:08:38.966030 17372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:38.966838 17387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:38.967011 17372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:08:38.967100 17372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268"
format_stamp: "Formatted at 2026-09-26 01:08:38 on dist-test-slave-l73v"
server_key: "cc0d7da4c0e106df893668022febb66d"
server_key_iv: "8225ef005552ffa91752a3f9716ee8d5"
server_key_version: "encryptionkey@0"
I20260926 01:08:38.967175 17372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:38.979363 17372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:38.981938 17372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:38.982096 17372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:38.986013 17372 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39273
I20260926 01:08:38.986047 17439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39273 every 8 connection(s)
I20260926 01:08:38.986403 17372 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:38.987223 17440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:38.989545 17440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268: Bootstrap starting.
I20260926 01:08:38.990110 17440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:38.990398 17440 log.cc:824] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:38.991173 17440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268: No bootstrap required, opened a new log
I20260926 01:08:38.991197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 17372
I20260926 01:08:38.991289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:38.991492 392 external_mini_cluster.cc:1428] Setting key e627578eeacb2cf5a31c422805c19c47
I20260926 01:08:38.993532 17440 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39273 } }
I20260926 01:08:38.993709 17440 raft_consensus.cc:374] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:38.993747 17440 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7c1e5a3eb74b9dbf3ffe100cec7268, State: Initialized, Role: FOLLOWER
I20260926 01:08:38.993877 17440 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39273 } }
I20260926 01:08:38.993964 17440 raft_consensus.cc:388] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:38.994004 17440 raft_consensus.cc:482] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:38.994046 17440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:38.994813 17440 raft_consensus.cc:504] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39273 } }
I20260926 01:08:38.994936 17440 leader_election.cc:302] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db7c1e5a3eb74b9dbf3ffe100cec7268; no voters:
I20260926 01:08:38.995103 17440 leader_election.cc:288] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:38.995147 17445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:38.995293 17445 raft_consensus.cc:686] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [term 1 LEADER]: Becoming Leader. State: Replica: db7c1e5a3eb74b9dbf3ffe100cec7268, State: Running, Role: LEADER
I20260926 01:08:38.995330 17440 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:38.995391 17445 consensus_queue.cc:237] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39273 } }
I20260926 01:08:38.995903 17447 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39273 } } }
I20260926 01:08:38.995981 17447 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:38.996133 17447 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db7c1e5a3eb74b9dbf3ffe100cec7268. Latest consensus state: current_term: 1 leader_uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c1e5a3eb74b9dbf3ffe100cec7268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39273 } } }
I20260926 01:08:38.996188 17447 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:38.996300 17454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:38.996937 17454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:38.998860 17454 catalog_manager.cc:1347] Generated new cluster ID: 6ab95e2ff01e4b2a9671e1b3c0f90d05
I20260926 01:08:38.998920 17454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:08:38 dist-test-slave-l73v krb5kdc[17359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:39.003113 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:38.991981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.0.98.62:39273)
0926 01:08:38.992127 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:38.992136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:38.992168 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:08:38.992755 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:38.992765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:38.992830 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:38.992988 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:38.993582 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:38.995248 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:38.996390 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:38.997008 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:38.997438 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:38.999827 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:38.999831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:38.999832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:38.999845 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:39.001441 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:39.001773 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:39.001776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:39.001777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:39.001863 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:39.002036 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:39.002037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:39.002038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:39.002282 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:39.002462 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.002588 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.002829 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260926 01:08:39.017673 17454 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:39.018455 17454 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:39.022531 17454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268: Generated new TSK 0
I20260926 01:08:39.022758 17454 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:08:39.054620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39273
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:39.137008 17466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:39.137238 17466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:39.137262 17466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:39.138568 17466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:39.138624 17466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:39.138655 17466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:08:39.140311 17466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:39.140580 17466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:39.140829 17466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:39.146729 17474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.146723 17471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.146724 17472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:39.146818 17466 server_base.cc:1034] running on GCE node
I20260926 01:08:39.147158 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:39.147406 17466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:39.148595 17466 hybrid_clock.cc:648] HybridClock initialized: now 1790384919148581 us; error 29 us; skew 500 ppm
Sep 26 01:08:39 dist-test-slave-l73v krb5kdc[17359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:39.151417 17466 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:08:39.151903 17466 webserver.cc:460] Webserver started at http://127.0.98.1:44559/ using document root <none> and password file <none>
I20260926 01:08:39.152226 17466 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:39.152300 17466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:39.152443 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:39.153550 17466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "63cb8fa10f0f490fb50ea8411f841d31"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "729cf66d2be5cb0a83455fce78dcb9d9"
server_key_iv: "c43e650dd2d4043eabbce3867b2f3f6d"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.153892 17466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "63cb8fa10f0f490fb50ea8411f841d31"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "729cf66d2be5cb0a83455fce78dcb9d9"
server_key_iv: "c43e650dd2d4043eabbce3867b2f3f6d"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.155320 17466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:39.156193 17481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.156428 17466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:39.156520 17466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "63cb8fa10f0f490fb50ea8411f841d31"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "729cf66d2be5cb0a83455fce78dcb9d9"
server_key_iv: "c43e650dd2d4043eabbce3867b2f3f6d"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.156589 17466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:39.166101 17466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:39.168437 17466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:39.168637 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:39.168880 17466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:39.169282 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:39.169348 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.169389 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:39.169412 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.175526 17466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34517
I20260926 01:08:39.175594 17594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34517 every 8 connection(s)
I20260926 01:08:39.175909 17466 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:08:39.179459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 17466
I20260926 01:08:39.179554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:08:39.179733 392 external_mini_cluster.cc:1428] Setting key 58b6dc4701cfe120a96f75e452f693f3
Sep 26 01:08:39 dist-test-slave-l73v krb5kdc[17359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:39.186668 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.176700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46561 (local address 127.0.98.62:39273)
0926 01:08:39.176765 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.176767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.176948 (+ 181us) server_negotiation.cc:407] Connection header received
0926 01:08:39.177959 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.177961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.177974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.178009 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:39.178760 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.179298 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.180660 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.180959 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.181044 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.183710 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:39.183712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:39.183712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:39.183721 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:39.185202 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:39.185650 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:39.185651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:39.185651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:39.185709 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:39.185954 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:39.185955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:39.185955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:39.186084 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:39.186147 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.186463 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.186565 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:08:39.186846 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.176650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39273 (local address 127.0.98.1:46561)
0926 01:08:39.176832 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:39.176869 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:39.177840 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:39.178109 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:39.178113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:39.178442 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:39.178665 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.178670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.179451 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.179452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.180574 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.180578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.181086 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.181087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.181216 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.181799 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:39.181808 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:39.183596 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:39.185295 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:39.185306 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:39.185309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:39.185571 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:39.185767 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:39.185768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:39.185768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:39.185895 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:39.186145 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:39.186152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:39.186305 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 01:08:39.186442 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 01:08:39.186616 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260926 01:08:39.187350 17595 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39273
I20260926 01:08:39.187469 17595 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:39.187674 17595 heartbeater.cc:511] Master 127.0.98.62:39273 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:08:39.188309 17404 ts_manager.cc:194] Registered new tserver with Master: 63cb8fa10f0f490fb50ea8411f841d31 (127.0.98.1:34517)
I20260926 01:08:39.189040 17404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:08:39.207139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39273
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:39.288522 17600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:39.288751 17600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:39.288777 17600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:39.290069 17600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:39.290132 17600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:39.290161 17600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:08:39.291915 17600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:39.292182 17600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:39.292438 17600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:39.297667 17608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.297839 17606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.297667 17605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:39.297935 17600 server_base.cc:1034] running on GCE node
I20260926 01:08:39.298161 17600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:39.298429 17600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:39.299606 17600 hybrid_clock.cc:648] HybridClock initialized: now 1790384919299555 us; error 65 us; skew 500 ppm
Sep 26 01:08:39 dist-test-slave-l73v krb5kdc[17359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:39.302183 17600 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:08:39.302707 17600 webserver.cc:460] Webserver started at http://127.0.98.2:41229/ using document root <none> and password file <none>
I20260926 01:08:39.302937 17600 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:39.302985 17600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:39.303113 17600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:39.304180 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "f5c0f29ffb5748d493201678a21866b5"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "187b5c7d2ea8cff4ac7e36cbf43c45eb"
server_key_iv: "2c99eb1d549a4ce02795dcc652335fc1"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.304507 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f5c0f29ffb5748d493201678a21866b5"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "187b5c7d2ea8cff4ac7e36cbf43c45eb"
server_key_iv: "2c99eb1d549a4ce02795dcc652335fc1"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.305873 17600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:08:39.306756 17615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.306973 17600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:08:39.307133 17600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "f5c0f29ffb5748d493201678a21866b5"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "187b5c7d2ea8cff4ac7e36cbf43c45eb"
server_key_iv: "2c99eb1d549a4ce02795dcc652335fc1"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.307220 17600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:39.319973 17600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:39.322062 17600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:39.322208 17600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:39.322501 17600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:39.322826 17600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:39.322862 17600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.322892 17600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:39.322909 17600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.328459 17600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41063
I20260926 01:08:39.328531 17728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41063 every 8 connection(s)
I20260926 01:08:39.328840 17600 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:08:39.332163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 17600
I20260926 01:08:39.332252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:08:39.332466 392 external_mini_cluster.cc:1428] Setting key 325176570482e5de86541ce1de166fc1
Sep 26 01:08:39 dist-test-slave-l73v krb5kdc[17359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:39.338781 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.329655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52187 (local address 127.0.98.62:39273)
0926 01:08:39.329736 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.329738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.329876 (+ 138us) server_negotiation.cc:407] Connection header received
0926 01:08:39.330898 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.330899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.330912 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.330959 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:39.331618 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.332124 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.333272 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.333473 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.333528 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.336095 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:39.336097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:39.336097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:39.336106 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:39.337415 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:39.337809 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:39.337811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:39.337811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:39.337864 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:39.338089 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:39.338090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:39.338091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:39.338208 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:39.338264 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.338591 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.338677 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 01:08:39.338961 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.329542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39273 (local address 127.0.98.2:52187)
0926 01:08:39.329740 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:39.329790 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:39.330781 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:39.331041 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:39.331044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:39.331319 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:39.331522 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.331526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.332215 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.332216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.333197 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.333200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.333546 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.333546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.333683 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.334323 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:39.334333 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:39.335988 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:39.337499 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:39.337509 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:39.337511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:39.337740 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:39.337919 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:39.337920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:39.337921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:39.338033 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:39.338270 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:39.338277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:39.338466 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 01:08:39.338563 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 01:08:39.338731 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
I20260926 01:08:39.339427 17729 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39273
I20260926 01:08:39.339545 17729 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:39.339759 17729 heartbeater.cc:511] Master 127.0.98.62:39273 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 01:08:39.340169 17404 ts_manager.cc:194] Registered new tserver with Master: f5c0f29ffb5748d493201678a21866b5 (127.0.98.2:41063)
I20260926 01:08:39.340546 17404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:08:39.358326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39273
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:39.441066 17734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:39.441308 17734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:39.441334 17734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:39.442572 17734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:39.442613 17734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:39.442641 17734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:08:39.444253 17734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:39.444481 17734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:39.444679 17734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:39.450578 17739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.450578 17740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:39.450598 17734 server_base.cc:1034] running on GCE node
W20260926 01:08:39.450582 17742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:39.451081 17734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:39.451370 17734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:39.452533 17734 hybrid_clock.cc:648] HybridClock initialized: now 1790384919452518 us; error 31 us; skew 500 ppm
Sep 26 01:08:39 dist-test-slave-l73v krb5kdc[17359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:39.455206 17734 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:08:39.455703 17734 webserver.cc:460] Webserver started at http://127.0.98.3:34979/ using document root <none> and password file <none>
I20260926 01:08:39.455910 17734 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:39.455955 17734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:39.456049 17734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:39.457077 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "842928c75a217a1062733e0f69c9d046"
server_key_iv: "7cf00f6b9e8f54787951ac9922fd6691"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.457389 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "842928c75a217a1062733e0f69c9d046"
server_key_iv: "7cf00f6b9e8f54787951ac9922fd6691"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.458791 17734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 01:08:39.459703 17749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.459904 17734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:39.460000 17734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "842928c75a217a1062733e0f69c9d046"
server_key_iv: "7cf00f6b9e8f54787951ac9922fd6691"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.460079 17734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:39.481068 17734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:39.483608 17734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:39.483740 17734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:39.483922 17734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:39.484186 17734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:39.484215 17734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.484236 17734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:39.484247 17734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.489717 17734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38869
I20260926 01:08:39.489800 17862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38869 every 8 connection(s)
I20260926 01:08:39.490094 17734 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:08:39.493671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 17734
I20260926 01:08:39.493803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:08:39.494006 392 external_mini_cluster.cc:1428] Setting key ae0302ed700b503a4859142543e3fa6c
Sep 26 01:08:39 dist-test-slave-l73v krb5kdc[17359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:39.500358 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.490895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50961 (local address 127.0.98.62:39273)
0926 01:08:39.490978 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.490981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.491160 (+ 179us) server_negotiation.cc:407] Connection header received
0926 01:08:39.492141 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.492143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.492156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.492224 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:39.492910 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.493431 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.494724 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.495023 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.495079 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.497273 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:39.497275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:39.497276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:39.497285 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:39.498618 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:39.499119 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:39.499120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:39.499121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:39.499187 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:39.499584 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:39.499585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:39.499586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:39.499738 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:39.499817 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.500180 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.500264 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 01:08:39.500520 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.490867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39273 (local address 127.0.98.3:50961)
0926 01:08:39.491040 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:39.491078 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:39.492026 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:39.492297 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:39.492301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:39.492586 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:39.492815 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.492820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.493524 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.493526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.494634 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.494638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.495095 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.495097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.495215 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.495692 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:39.495701 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:39.497155 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:39.498724 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:39.498733 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:39.498735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:39.499002 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:39.499270 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:39.499271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:39.499271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:39.499444 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:39.499886 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:39.499894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:39.500044 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 01:08:39.500143 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 01:08:39.500303 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260926 01:08:39.501020 17863 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39273
I20260926 01:08:39.501120 17863 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:39.501317 17863 heartbeater.cc:511] Master 127.0.98.62:39273 requested a full tablet report, sending...
I20260926 01:08:39.501776 17404 ts_manager.cc:194] Registered new tserver with Master: cfd1470a5d6a4eb58e45e6ec845be3f3 (127.0.98.3:38869)
I20260926 01:08:39.502182 17404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50961
I20260926 01:08:39.505975 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:08:39.513957 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.507382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.0.98.62:39273)
0926 01:08:39.507446 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.507448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.507598 (+ 150us) server_negotiation.cc:407] Connection header received
0926 01:08:39.507630 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.507631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.507642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.507687 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:39.508359 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.508856 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.509634 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.509832 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.509899 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.511096 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:39.511098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:39.511098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:39.511106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:39.512591 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:39.512982 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:39.512983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:39.512984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:39.513044 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:39.513318 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:39.513319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:39.513320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:39.513491 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:39.513578 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.513762 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.513851 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 01:08:39.515637 17404 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:08:39.517203 17404 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:08:39.523893 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.519630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41063 (local address 127.0.0.1:53998)
0926 01:08:39.519984 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:39.519992 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:39.520025 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:39.520568 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:39.520568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:39.520570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:39.520606 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.520607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.521496 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.521497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.522761 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.522764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.523687 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.523688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.523744 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.523747 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:39.523793 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:08:39.523823 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":79,"threads_started":1}
I20260926 01:08:39.523939 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.519848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.0.98.2:41063)
0926 01:08:39.520412 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.520417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.520425 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:08:39.520460 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.520462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.520476 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.520671 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:39.520754 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.521398 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.522858 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.523586 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.523677 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.523737 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.523811 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.523855 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":101,"threads_started":1}
I20260926 01:08:39.523893 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.519629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34517 (local address 127.0.0.1:46524)
0926 01:08:39.519759 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:39.519814 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:39.519883 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:39.520275 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:39.520275 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:39.520278 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:39.520354 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.520356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.521768 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.521769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.522861 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.522862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.523560 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.523561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.523650 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.523665 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 01:08:39.523733 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 01:08:39.523792 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"spinlock_wait_cycles":896,"thread_start_us":81,"threads_started":1}
I20260926 01:08:39.524597 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.519629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38869 (local address 127.0.0.1:48304)
0926 01:08:39.520041 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:39.520051 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:39.520107 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:39.520257 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:39.520260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:39.520262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:39.520344 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.520346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.522469 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.522470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.523546 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.523549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.524367 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.524369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.524431 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.524435 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:39.524482 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 01:08:39.524519 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260926 01:08:39.524628 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.519907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.0.98.1:34517)
0926 01:08:39.520075 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.520080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.520090 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:08:39.520133 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.520135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.520151 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.520267 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:39.520488 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.521665 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.522933 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.523481 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.523548 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.523612 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.523941 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.524550 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260926 01:08:39.524765 17663 tablet_service.cc:1467] Processing CreateTablet for tablet fbe9b7c4d76c4b19bafbc11321bb803f (DEFAULT_TABLE table=test-table [id=e19b3e5f0e7744ab85703be16c3ff54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:39.525080 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.519888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.0.98.3:38869)
0926 01:08:39.520118 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.520122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.520130 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:08:39.520160 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.520162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.520174 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.520241 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:39.520407 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.521656 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.523650 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.524251 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.524364 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.524420 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.524952 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.524998 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260926 01:08:39.525147 17663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe9b7c4d76c4b19bafbc11321bb803f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:39.525229 17529 tablet_service.cc:1467] Processing CreateTablet for tablet fbe9b7c4d76c4b19bafbc11321bb803f (DEFAULT_TABLE table=test-table [id=e19b3e5f0e7744ab85703be16c3ff54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:39.525532 17529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe9b7c4d76c4b19bafbc11321bb803f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:39.525640 17797 tablet_service.cc:1467] Processing CreateTablet for tablet fbe9b7c4d76c4b19bafbc11321bb803f (DEFAULT_TABLE table=test-table [id=e19b3e5f0e7744ab85703be16c3ff54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:39.525929 17797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe9b7c4d76c4b19bafbc11321bb803f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:39.528439 17877 tablet_bootstrap.cc:492] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: Bootstrap starting.
I20260926 01:08:39.528874 17878 tablet_bootstrap.cc:492] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31: Bootstrap starting.
I20260926 01:08:39.528875 17879 tablet_bootstrap.cc:492] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: Bootstrap starting.
I20260926 01:08:39.529201 17877 tablet_bootstrap.cc:654] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:39.529508 17878 tablet_bootstrap.cc:654] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:39.529508 17879 tablet_bootstrap.cc:654] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:39.529680 17877 log.cc:824] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:39.530057 17878 log.cc:824] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:39.530057 17879 log.cc:824] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:39.530417 17877 tablet_bootstrap.cc:492] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: No bootstrap required, opened a new log
I20260926 01:08:39.530491 17877 ts_tablet_manager.cc:1397] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:08:39.530869 17879 tablet_bootstrap.cc:492] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: No bootstrap required, opened a new log
I20260926 01:08:39.530938 17879 ts_tablet_manager.cc:1397] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:08:39.531097 17878 tablet_bootstrap.cc:492] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31: No bootstrap required, opened a new log
I20260926 01:08:39.531172 17878 ts_tablet_manager.cc:1397] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:08:39.532400 17877 raft_consensus.cc:348] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 } } peers { permanent_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38869 } } peers { permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 } }
I20260926 01:08:39.532606 17877 raft_consensus.cc:374] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:39.532641 17877 raft_consensus.cc:729] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd1470a5d6a4eb58e45e6ec845be3f3, State: Initialized, Role: FOLLOWER
I20260926 01:08:39.532655 17879 raft_consensus.cc:348] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 } } peers { permanent_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38869 } } peers { permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 } }
I20260926 01:08:39.532797 17877 consensus_queue.cc:260] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 } } peers { permanent_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38869 } } peers { permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 } }
I20260926 01:08:39.532846 17879 raft_consensus.cc:374] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:39.532889 17879 raft_consensus.cc:729] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5c0f29ffb5748d493201678a21866b5, State: Initialized, Role: FOLLOWER
I20260926 01:08:39.533031 17879 consensus_queue.cc:260] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 } } peers { permanent_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38869 } } peers { permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 } }
I20260926 01:08:39.533100 17877 ts_tablet_manager.cc:1428] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:39.533134 17878 raft_consensus.cc:348] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 } } peers { permanent_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38869 } } peers { permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 } }
I20260926 01:08:39.533252 17863 heartbeater.cc:503] Master 127.0.98.62:39273 was elected leader, sending a full tablet report...
I20260926 01:08:39.533303 17879 ts_tablet_manager.cc:1428] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 01:08:39.533301 17878 raft_consensus.cc:374] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:39.533344 17878 raft_consensus.cc:729] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cb8fa10f0f490fb50ea8411f841d31, State: Initialized, Role: FOLLOWER
I20260926 01:08:39.533414 17729 heartbeater.cc:503] Master 127.0.98.62:39273 was elected leader, sending a full tablet report...
I20260926 01:08:39.533442 17878 consensus_queue.cc:260] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 } } peers { permanent_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38869 } } peers { permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 } }
I20260926 01:08:39.533787 17878 ts_tablet_manager.cc:1428] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:39.533810 17595 heartbeater.cc:503] Master 127.0.98.62:39273 was elected leader, sending a full tablet report...
I20260926 01:08:39.540117 17883 raft_consensus.cc:482] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:08:39.540239 17883 raft_consensus.cc:504] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 } } peers { permanent_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38869 } } peers { permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 } }
I20260926 01:08:39.540618 17883 leader_election.cc:288] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5c0f29ffb5748d493201678a21866b5 (127.0.98.2:41063), 63cb8fa10f0f490fb50ea8411f841d31 (127.0.98.1:34517)
I20260926 01:08:39.543955 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.540936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57283 (local address 127.0.98.1:34517)
0926 01:08:39.540983 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.540985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.541246 (+ 261us) server_negotiation.cc:407] Connection header received
0926 01:08:39.541292 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.541293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.541304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.541357 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:39.541466 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.542142 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.543183 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.543623 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.543689 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.543700 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.543839 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.543880 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:08:39.543998 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.540824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34517 (local address 127.0.98.3:57283)
0926 01:08:39.541156 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:39.541165 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:39.541240 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:39.541357 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:39.541358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:39.541359 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:39.541414 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.541415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.542229 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.542230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.543087 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.543090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.543698 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.543699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.543779 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.543783 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:39.543828 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:08:39.543873 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":142,"thread_start_us":118,"threads_started":1}
I20260926 01:08:39.544011 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.540958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34827 (local address 127.0.98.2:41063)
0926 01:08:39.541007 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.541010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.541019 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:08:39.541047 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.541048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.541060 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.541115 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:39.541345 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.542099 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.543207 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.543670 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.543735 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.543746 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.543902 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.543944 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:08:39.544327 17549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe9b7c4d76c4b19bafbc11321bb803f" candidate_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63cb8fa10f0f490fb50ea8411f841d31" is_pre_election: true
I20260926 01:08:39.544507 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.540824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41063 (local address 127.0.98.3:34827)
0926 01:08:39.540863 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:39.540879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:39.540920 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:39.541137 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:39.541138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:39.541141 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:39.541223 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.541226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.542186 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.542187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.543114 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.543117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.543747 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.543748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.543840 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.543843 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:39.543887 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:08:39.543928 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 01:08:39.544500 17549 raft_consensus.cc:2454] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfd1470a5d6a4eb58e45e6ec845be3f3 in term 0.
I20260926 01:08:39.544740 17750 leader_election.cc:302] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63cb8fa10f0f490fb50ea8411f841d31, cfd1470a5d6a4eb58e45e6ec845be3f3; no voters:
I20260926 01:08:39.544885 17883 raft_consensus.cc:2781] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:08:39.544894 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe9b7c4d76c4b19bafbc11321bb803f" candidate_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c0f29ffb5748d493201678a21866b5" is_pre_election: true
I20260926 01:08:39.544970 17883 raft_consensus.cc:482] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:08:39.544999 17883 raft_consensus.cc:3037] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:39.545034 17683 raft_consensus.cc:2454] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfd1470a5d6a4eb58e45e6ec845be3f3 in term 0.
I20260926 01:08:39.545782 17883 raft_consensus.cc:504] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 } } peers { permanent_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38869 } } peers { permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 } }
I20260926 01:08:39.545908 17883 leader_election.cc:288] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [CANDIDATE]: Term 1 election: Requested vote from peers f5c0f29ffb5748d493201678a21866b5 (127.0.98.2:41063), 63cb8fa10f0f490fb50ea8411f841d31 (127.0.98.1:34517)
I20260926 01:08:39.546060 17683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe9b7c4d76c4b19bafbc11321bb803f" candidate_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c0f29ffb5748d493201678a21866b5"
I20260926 01:08:39.546072 17549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe9b7c4d76c4b19bafbc11321bb803f" candidate_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63cb8fa10f0f490fb50ea8411f841d31"
I20260926 01:08:39.546144 17683 raft_consensus.cc:3037] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:39.546152 17549 raft_consensus.cc:3037] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:39.546893 17683 raft_consensus.cc:2454] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfd1470a5d6a4eb58e45e6ec845be3f3 in term 1.
I20260926 01:08:39.546996 17549 raft_consensus.cc:2454] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfd1470a5d6a4eb58e45e6ec845be3f3 in term 1.
I20260926 01:08:39.547123 17751 leader_election.cc:302] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfd1470a5d6a4eb58e45e6ec845be3f3, f5c0f29ffb5748d493201678a21866b5; no voters:
I20260926 01:08:39.547215 17883 raft_consensus.cc:2781] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:39.547371 17883 raft_consensus.cc:686] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 1 LEADER]: Becoming Leader. State: Replica: cfd1470a5d6a4eb58e45e6ec845be3f3, State: Running, Role: LEADER
I20260926 01:08:39.547457 17883 consensus_queue.cc:237] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 } } peers { permanent_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38869 } } peers { permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 } }
I20260926 01:08:39.548384 17404 catalog_manager.cc:5495] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfd1470a5d6a4eb58e45e6ec845be3f3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfd1470a5d6a4eb58e45e6ec845be3f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 } health_report { overall_health: UNKNOWN } } }
I20260926 01:08:39.553349 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.551167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.0.98.3:38869)
0926 01:08:39.551209 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.551210 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.551220 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:08:39.551283 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.551284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.551303 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.551358 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:08:39.551468 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.551948 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.552684 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.552898 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.552953 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.553077 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:08:39.553177 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:08:39.553231 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.553248 (+ 17us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.553268 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:08:39.555282 17683 raft_consensus.cc:1261] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5 [term 1 FOLLOWER]: Refusing update from remote peer cfd1470a5d6a4eb58e45e6ec845be3f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:39.555624 17549 raft_consensus.cc:1261] T fbe9b7c4d76c4b19bafbc11321bb803f P 63cb8fa10f0f490fb50ea8411f841d31 [term 1 FOLLOWER]: Refusing update from remote peer cfd1470a5d6a4eb58e45e6ec845be3f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:39.555689 17887 consensus_queue.cc:1035] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5c0f29ffb5748d493201678a21866b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:39.555868 17883 consensus_queue.cc:1035] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63cb8fa10f0f490fb50ea8411f841d31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:39.558589 17890 mvcc.cc:205] Tried to move back new op lower bound from 7333416630495772672 to 7333416630467313664. Current Snapshot: MvccSnapshot[applied={T|T < 7333416630495772672}]
I20260926 01:08:39.559197 17891 mvcc.cc:205] Tried to move back new op lower bound from 7333416630495772672 to 7333416630467313664. Current Snapshot: MvccSnapshot[applied={T|T < 7333416630495772672 or (T in {7333416630495772672})}]
I20260926 01:08:39.559473 17892 mvcc.cc:205] Tried to move back new op lower bound from 7333416630495772672 to 7333416630467313664. Current Snapshot: MvccSnapshot[applied={T|T < 7333416630495772672}]
I20260926 01:08:39.561482 17403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:39.561597 17403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:39.562530 17403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db7c1e5a3eb74b9dbf3ffe100cec7268: Sending DeleteTablet for 3 replicas of tablet fbe9b7c4d76c4b19bafbc11321bb803f
I20260926 01:08:39.562942 17663 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe9b7c4d76c4b19bafbc11321bb803f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53998
I20260926 01:08:39.563040 17797 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe9b7c4d76c4b19bafbc11321bb803f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48304
I20260926 01:08:39.563282 17901 tablet_replica.cc:330] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: stopping tablet replica
I20260926 01:08:39.563395 17529 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe9b7c4d76c4b19bafbc11321bb803f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46524
I20260926 01:08:39.563433 17901 raft_consensus.cc:2229] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:08:39.563517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 17466
I20260926 01:08:39.563668 17901 raft_consensus.cc:2258] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:39.565528 17900 tablet_replica.cc:330] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: stopping tablet replica
I20260926 01:08:39.565676 17900 raft_consensus.cc:2229] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:08:39.565809 17900 raft_consensus.cc:2258] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:39.566175 17901 ts_tablet_manager.cc:1905] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:39.566246 17900 ts_tablet_manager.cc:1905] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:39.568495 17900 ts_tablet_manager.cc:1918] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:39.568594 17900 log.cc:1196] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/fbe9b7c4d76c4b19bafbc11321bb803f
I20260926 01:08:39.568516 17901 ts_tablet_manager.cc:1918] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:39.568784 17901 log.cc:1196] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/fbe9b7c4d76c4b19bafbc11321bb803f
I20260926 01:08:39.568899 17900 ts_tablet_manager.cc:1939] T fbe9b7c4d76c4b19bafbc11321bb803f P f5c0f29ffb5748d493201678a21866b5: Deleting consensus metadata
I20260926 01:08:39.569159 17901 ts_tablet_manager.cc:1939] T fbe9b7c4d76c4b19bafbc11321bb803f P cfd1470a5d6a4eb58e45e6ec845be3f3: Deleting consensus metadata
I20260926 01:08:39.569427 17389 catalog_manager.cc:4841] TS f5c0f29ffb5748d493201678a21866b5 (127.0.98.2:41063): tablet fbe9b7c4d76c4b19bafbc11321bb803f (table test-table [id=e19b3e5f0e7744ab85703be16c3ff54e]) successfully deleted
I20260926 01:08:39.569590 17391 catalog_manager.cc:4841] TS cfd1470a5d6a4eb58e45e6ec845be3f3 (127.0.98.3:38869): tablet fbe9b7c4d76c4b19bafbc11321bb803f (table test-table [id=e19b3e5f0e7744ab85703be16c3ff54e]) successfully deleted
W20260926 01:08:39.569706 17388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34517 (error 108)
I20260926 01:08:39.570019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 17600
I20260926 01:08:39.570573 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.570102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34517 (local address 127.0.0.1:46534)
0926 01:08:39.570498 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:39.570538 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374}
W20260926 01:08:39.574354 17388 catalog_manager.cc:4584] TS 63cb8fa10f0f490fb50ea8411f841d31 (127.0.98.1:34517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbe9b7c4d76c4b19bafbc11321bb803f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34517: connect: Connection refused (error 111)
I20260926 01:08:39.574976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 17734
I20260926 01:08:39.579435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 17372
2026-09-26T01:08:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1794 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T01:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:39Z Disabled control of system clock
I20260926 01:08:39.605630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32929
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:08:39.687331 17907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:39.687582 17907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:39.687603 17907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:39.688894 17907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:39.688951 17907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:39.688962 17907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:39.688974 17907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:39.688985 17907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:39.690579 17907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:39.690852 17907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:39.691170 17907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:39.696027 17915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.696048 17912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.696035 17913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:39.696321 17907 server_base.cc:1034] running on GCE node
I20260926 01:08:39.696560 17907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:39.696857 17907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:39.698027 17907 hybrid_clock.cc:648] HybridClock initialized: now 1790384919698005 us; error 34 us; skew 500 ppm
I20260926 01:08:39.699515 17907 webserver.cc:460] Webserver started at http://127.0.98.62:33457/ using document root <none> and password file <none>
I20260926 01:08:39.699776 17907 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:39.699853 17907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:39.699985 17907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:39.701112 17907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "4fdbbc40ce1b4369867ea40c7672345f"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "1fc657d937802f63adfc99f78ea13b66"
server_key_iv: "a32c6c47338af00d2d9fdeb83657e434"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.701471 17907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "4fdbbc40ce1b4369867ea40c7672345f"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "1fc657d937802f63adfc99f78ea13b66"
server_key_iv: "a32c6c47338af00d2d9fdeb83657e434"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.702826 17907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 01:08:39.703723 17921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.703938 17907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:39.704030 17907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "4fdbbc40ce1b4369867ea40c7672345f"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "1fc657d937802f63adfc99f78ea13b66"
server_key_iv: "a32c6c47338af00d2d9fdeb83657e434"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.704097 17907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:39.720695 17907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:39.721181 17907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:39.721344 17907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:39.725868 17907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32929
I20260926 01:08:39.725905 17973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32929 every 8 connection(s)
I20260926 01:08:39.726269 17907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:39.727384 17974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:39.729938 17974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f: Bootstrap starting.
I20260926 01:08:39.730548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 17907
I20260926 01:08:39.730640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:39.730693 17974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:39.730849 392 external_mini_cluster.cc:1428] Setting key 35ec7df31daa054987d6b3dda48b114c
I20260926 01:08:39.731032 17974 log.cc:824] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:39.731868 17974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f: No bootstrap required, opened a new log
I20260926 01:08:39.734247 17974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbbc40ce1b4369867ea40c7672345f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32929 } }
I20260926 01:08:39.734622 17974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:39.734673 17974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdbbc40ce1b4369867ea40c7672345f, State: Initialized, Role: FOLLOWER
I20260926 01:08:39.734808 17974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbbc40ce1b4369867ea40c7672345f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32929 } }
I20260926 01:08:39.734891 17974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:39.734933 17974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:39.735038 17974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:39.735991 17974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbbc40ce1b4369867ea40c7672345f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32929 } }
I20260926 01:08:39.736096 17974 leader_election.cc:302] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fdbbc40ce1b4369867ea40c7672345f; no voters:
I20260926 01:08:39.736414 17974 leader_election.cc:288] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:39.736485 17979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:39.736706 17979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [term 1 LEADER]: Becoming Leader. State: Replica: 4fdbbc40ce1b4369867ea40c7672345f, State: Running, Role: LEADER
I20260926 01:08:39.736737 17974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:39.736814 17979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbbc40ce1b4369867ea40c7672345f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32929 } }
I20260926 01:08:39.737260 17980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fdbbc40ce1b4369867ea40c7672345f. Latest consensus state: current_term: 1 leader_uuid: "4fdbbc40ce1b4369867ea40c7672345f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbbc40ce1b4369867ea40c7672345f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32929 } } }
I20260926 01:08:39.737349 17980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:39.737452 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.731399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.0.98.62:32929)
0926 01:08:39.731596 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.731603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.731630 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:08:39.732136 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.732144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.732208 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.732335 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:39.732619 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.733661 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.734945 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.735443 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.735542 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.736487 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:39.736492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:39.736492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:39.736710 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:39.736759 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:39.736776 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:39.736843 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.737001 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.737140 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260926 01:08:39.737555 17988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:39.737219 17981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fdbbc40ce1b4369867ea40c7672345f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdbbc40ce1b4369867ea40c7672345f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32929 } } }
I20260926 01:08:39.738129 17981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:39.738530 17988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:39.740286 17988 catalog_manager.cc:1347] Generated new cluster ID: 3f144443fad84c02a901ff211040eeed
I20260926 01:08:39.740339 17988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:39.753779 17988 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:39.754359 17988 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:39.757349 17988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f: Generated new TSK 0
I20260926 01:08:39.757576 17988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:39.762269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32929
--builtin_ntp_servers=127.0.98.20:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:08:39.854175 17998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:39.854429 17998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:39.854465 17998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:39.855752 17998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:39.855800 17998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:39.855836 17998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:08:39.857376 17998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:39.857650 17998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:39.857965 17998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:39.863183 18004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.863170 18003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.863165 18006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:39.863631 17998 server_base.cc:1034] running on GCE node
I20260926 01:08:39.863875 17998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:39.864141 17998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:39.865319 17998 hybrid_clock.cc:648] HybridClock initialized: now 1790384919865293 us; error 42 us; skew 500 ppm
I20260926 01:08:39.866926 17998 webserver.cc:460] Webserver started at http://127.0.98.1:46745/ using document root <none> and password file <none>
I20260926 01:08:39.867239 17998 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:39.867290 17998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:39.867419 17998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:39.868587 17998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "4159f55bc42f4aa39704f90411e9973d"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "e992998f15da4d3bced816733e7041df"
server_key_iv: "0e44c0174540f36eb14936b5558cf93c"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.868927 17998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4159f55bc42f4aa39704f90411e9973d"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "e992998f15da4d3bced816733e7041df"
server_key_iv: "0e44c0174540f36eb14936b5558cf93c"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.870234 17998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:39.871017 18012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.871169 17998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:08:39.871255 17998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "4159f55bc42f4aa39704f90411e9973d"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "e992998f15da4d3bced816733e7041df"
server_key_iv: "0e44c0174540f36eb14936b5558cf93c"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.871333 17998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:39.890096 17998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:39.890645 17998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:39.890830 17998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:39.891101 17998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:39.891438 17998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:39.891484 17998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.891513 17998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:39.891539 17998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:39.897243 17998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41485
I20260926 01:08:39.897296 18125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41485 every 8 connection(s)
I20260926 01:08:39.897655 17998 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:08:39.898313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 17998
I20260926 01:08:39.898433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:08:39.898653 392 external_mini_cluster.cc:1428] Setting key c3b8b3a53ff06711e4f23c59145a6bf5
I20260926 01:08:39.899612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32929
--builtin_ntp_servers=127.0.98.20:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:08:39.903825 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.898518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33897 (local address 127.0.98.62:32929)
0926 01:08:39.898575 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:39.898578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:39.898868 (+ 290us) server_negotiation.cc:407] Connection header received
0926 01:08:39.899383 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:39.899385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:39.899398 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:39.899454 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:39.900196 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.900830 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.902197 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.902451 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.902520 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.903513 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:39.903515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:39.903515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:39.903522 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:39.903546 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:39.903551 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:39.903599 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:39.903728 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 01:08:39.903757 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:08:39.904090 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:39.898461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32929 (local address 127.0.98.1:33897)
0926 01:08:39.898715 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:39.898766 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:39.899290 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:39.899537 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:39.899540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:39.899549 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:39.900105 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.900111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.900958 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.900960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.902093 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:39.902096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:39.902533 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:39.902534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:39.902704 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:39.903332 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:08:39.903333 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:39.903378 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:08:39.903413 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:39.903617 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:39.903622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:39.903626 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:39.903700 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 01:08:39.903826 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":132,"threads_started":1}
I20260926 01:08:39.904721 18126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32929
I20260926 01:08:39.904841 18126 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:39.905078 18126 heartbeater.cc:511] Master 127.0.98.62:32929 requested a full tablet report, sending...
I20260926 01:08:39.905705 17938 ts_manager.cc:194] Registered new tserver with Master: 4159f55bc42f4aa39704f90411e9973d (127.0.98.1:41485)
I20260926 01:08:39.906364 17938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33897
W20260926 01:08:39.984628 18129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:39.984854 18129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:39.984879 18129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:39.986177 18129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:39.986229 18129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:39.986258 18129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:08:39.987766 18129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:39.988023 18129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:39.988341 18129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:39.993331 18135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.993331 18134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:39.993495 18137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:39.993398 18129 server_base.cc:1034] running on GCE node
I20260926 01:08:39.993729 18129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:39.993932 18129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:39.995062 18129 hybrid_clock.cc:648] HybridClock initialized: now 1790384919995052 us; error 28 us; skew 500 ppm
I20260926 01:08:39.996261 18129 webserver.cc:460] Webserver started at http://127.0.98.2:42821/ using document root <none> and password file <none>
I20260926 01:08:39.996496 18129 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:39.996546 18129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:39.996670 18129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:39.997705 18129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "412ff5721377485e9457cbf67471d890"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "4b12e7f233e61a1cc4819a0176105f7a"
server_key_iv: "3c78736f9e05654a35bcbb90ac588889"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.998029 18129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "412ff5721377485e9457cbf67471d890"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "4b12e7f233e61a1cc4819a0176105f7a"
server_key_iv: "3c78736f9e05654a35bcbb90ac588889"
server_key_version: "encryptionkey@0"
I20260926 01:08:39.999287 18129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:08:40.000038 18143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:40.000211 18129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:08:40.000305 18129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "412ff5721377485e9457cbf67471d890"
format_stamp: "Formatted at 2026-09-26 01:08:39 on dist-test-slave-l73v"
server_key: "4b12e7f233e61a1cc4819a0176105f7a"
server_key_iv: "3c78736f9e05654a35bcbb90ac588889"
server_key_version: "encryptionkey@0"
I20260926 01:08:40.000372 18129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:40.045359 18129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:40.045687 18129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:40.045818 18129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:40.046049 18129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:40.046357 18129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:40.046427 18129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:40.046466 18129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:40.046485 18129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:40.052109 18129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43787
I20260926 01:08:40.052158 18256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43787 every 8 connection(s)
I20260926 01:08:40.052495 18129 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:08:40.055480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 18129
I20260926 01:08:40.055579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:08:40.055778 392 external_mini_cluster.cc:1428] Setting key 6138cdd819cc3036eeabb02b5c3a7550
I20260926 01:08:40.056708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32929
--builtin_ntp_servers=127.0.98.20:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:08:40.058099 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.053280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58309 (local address 127.0.98.62:32929)
0926 01:08:40.053361 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:40.053363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:40.053513 (+ 150us) server_negotiation.cc:407] Connection header received
0926 01:08:40.054034 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:40.054035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:40.054048 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:40.054095 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:40.054831 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.055353 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.056565 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.056765 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.056824 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.057822 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:40.057823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:40.057824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:40.057830 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:40.057846 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:40.057850 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:40.057893 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:40.057975 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 01:08:40.058007 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 01:08:40.058326 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.053179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32929 (local address 127.0.98.2:58309)
0926 01:08:40.053390 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:40.053431 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:40.053933 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:40.054179 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:40.054182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:40.054192 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:40.054741 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.054747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.055450 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.055451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.056483 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.056487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.056846 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.056847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.056998 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.057643 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:08:40.057644 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:40.057676 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:08:40.057704 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:40.057903 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:40.057909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:40.057913 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:40.057979 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 01:08:40.058097 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260926 01:08:40.058828 18257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32929
I20260926 01:08:40.058925 18257 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:40.059134 18257 heartbeater.cc:511] Master 127.0.98.62:32929 requested a full tablet report, sending...
I20260926 01:08:40.059500 17938 ts_manager.cc:194] Registered new tserver with Master: 412ff5721377485e9457cbf67471d890 (127.0.98.2:43787)
I20260926 01:08:40.059887 17938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58309
W20260926 01:08:40.138454 18260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:40.138682 18260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:40.138706 18260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:40.139990 18260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:40.140044 18260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:40.140074 18260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:08:40.141547 18260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:40.141816 18260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:40.142143 18260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:40.147195 18266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:40.147241 18265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:40.147377 18260 server_base.cc:1034] running on GCE node
W20260926 01:08:40.147195 18268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:40.147688 18260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:40.147958 18260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:40.149142 18260 hybrid_clock.cc:648] HybridClock initialized: now 1790384920149126 us; error 34 us; skew 500 ppm
I20260926 01:08:40.150430 18260 webserver.cc:460] Webserver started at http://127.0.98.3:33857/ using document root <none> and password file <none>
I20260926 01:08:40.150679 18260 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:40.150730 18260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:40.150858 18260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:40.151937 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "a4d64c053a904d6b9a6b5f80a7e03a01"
format_stamp: "Formatted at 2026-09-26 01:08:40 on dist-test-slave-l73v"
server_key: "6be297cad8396250174cde7214fe0b81"
server_key_iv: "7b1900cd8e6e7c1716ffee0a1b5efacc"
server_key_version: "encryptionkey@0"
I20260926 01:08:40.152319 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a4d64c053a904d6b9a6b5f80a7e03a01"
format_stamp: "Formatted at 2026-09-26 01:08:40 on dist-test-slave-l73v"
server_key: "6be297cad8396250174cde7214fe0b81"
server_key_iv: "7b1900cd8e6e7c1716ffee0a1b5efacc"
server_key_version: "encryptionkey@0"
I20260926 01:08:40.153620 18260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:40.154552 18274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:40.154711 18260 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:40.154803 18260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "a4d64c053a904d6b9a6b5f80a7e03a01"
format_stamp: "Formatted at 2026-09-26 01:08:40 on dist-test-slave-l73v"
server_key: "6be297cad8396250174cde7214fe0b81"
server_key_iv: "7b1900cd8e6e7c1716ffee0a1b5efacc"
server_key_version: "encryptionkey@0"
I20260926 01:08:40.154877 18260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:40.181921 18260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:40.182242 18260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:40.182433 18260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:40.182680 18260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:40.183014 18260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:40.183051 18260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:40.183079 18260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:40.183097 18260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:40.188803 18260 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42785
I20260926 01:08:40.188890 18387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42785 every 8 connection(s)
I20260926 01:08:40.189189 18260 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:08:40.192746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 18260
I20260926 01:08:40.192971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:08:40.193187 392 external_mini_cluster.cc:1428] Setting key 41c8bde0f213487a3d66f4583ed421ab
I20260926 01:08:40.194957 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.189960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37811 (local address 127.0.98.62:32929)
0926 01:08:40.190022 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:40.190024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:40.190333 (+ 309us) server_negotiation.cc:407] Connection header received
0926 01:08:40.190924 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:40.190925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:40.190937 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:40.190993 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:40.191691 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.192200 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.193434 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.193715 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.193774 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.194727 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:40.194728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:40.194728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:40.194733 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:40.194749 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:40.194753 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:40.194798 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:40.194865 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 01:08:40.194892 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:08:40.195180 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.189975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32929 (local address 127.0.98.3:37811)
0926 01:08:40.190186 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:40.190222 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:40.190766 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:40.191073 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:40.191076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:40.191086 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:40.191593 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.191597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.192291 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.192293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.193339 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.193342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.193796 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.193797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.193951 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.194584 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:08:40.194585 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:40.194618 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:08:40.194644 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:40.194801 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:40.194803 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:40.194805 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 01:08:40.194861 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:08:40.194951 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260926 01:08:40.195593 18388 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32929
I20260926 01:08:40.195684 18388 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:40.195873 18388 heartbeater.cc:511] Master 127.0.98.62:32929 requested a full tablet report, sending...
I20260926 01:08:40.196233 17938 ts_manager.cc:194] Registered new tserver with Master: a4d64c053a904d6b9a6b5f80a7e03a01 (127.0.98.3:42785)
I20260926 01:08:40.196620 17938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37811
I20260926 01:08:40.205272 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:08:40.209640 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.206717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.0.98.62:32929)
0926 01:08:40.206765 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:40.206767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:40.206801 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:08:40.206867 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:40.206868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:40.206877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:40.206935 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:40.207090 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.207647 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.208433 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.208645 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.208709 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.209382 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:40.209383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:40.209383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:40.209389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:40.209404 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:40.209407 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:40.209451 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:40.209549 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 01:08:40.209573 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:08:40.211336 17938 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:08:40.212953 17938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:08:40.219327 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.215406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.0.98.2:43787)
0926 01:08:40.216097 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:40.216101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:40.216110 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:08:40.216144 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:40.216146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:40.216160 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:40.216257 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:40.216346 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.217326 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.218200 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.218910 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.218972 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.219072 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:40.219190 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 01:08:40.219228 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":85,"threads_started":1}
I20260926 01:08:40.219355 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.215284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43787 (local address 127.0.0.1:35748)
0926 01:08:40.215567 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:40.215576 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:40.215615 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:40.216256 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:40.216257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:40.216258 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:40.216292 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.216294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.217401 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.217402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.218116 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.218118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.218986 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.218987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.219086 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.219103 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 01:08:40.219169 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 01:08:40.219229 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":71,"threads_started":1}
I20260926 01:08:40.219537 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.215676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.0.98.3:42785)
0926 01:08:40.215860 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:40.215865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:40.215978 (+ 113us) server_negotiation.cc:407] Connection header received
0926 01:08:40.216036 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:40.216038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:40.216050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:40.216113 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:40.216212 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.217102 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.218187 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.218954 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.219023 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.219105 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:40.219397 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 01:08:40.219442 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260926 01:08:40.219682 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.215494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42785 (local address 127.0.0.1:37502)
0926 01:08:40.215909 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:40.215919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:40.215976 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:40.216117 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:40.216118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:40.216120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:40.216156 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.216157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.217174 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.217175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.218093 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.218096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.219138 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.219140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.219199 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.219202 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:40.219245 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 01:08:40.219286 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":53,"threads_started":1}
I20260926 01:08:40.220394 18322 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce723a23a03497e907b58e878d9ba3d (DEFAULT_TABLE table=test-table [id=dc8e874aa93a4cd8a81332e746ffd875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:40.220566 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.215284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41485 (local address 127.0.0.1:56964)
0926 01:08:40.215466 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:40.215517 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:40.215590 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:40.215800 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:40.215803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:40.215805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:40.215880 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.215883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.218189 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.218190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.219337 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.219339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.220316 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.220317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.220381 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.220385 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:40.220441 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 01:08:40.220485 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260926 01:08:40.220577 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.215382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.0.98.1:41485)
0926 01:08:40.215621 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:40.215627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:40.215639 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:08:40.215681 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:40.215683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:40.215698 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:40.215795 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:40.215958 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.217095 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.219429 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.220237 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.220303 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.220393 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:40.220455 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 01:08:40.220493 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260926 01:08:40.220490 18191 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce723a23a03497e907b58e878d9ba3d (DEFAULT_TABLE table=test-table [id=dc8e874aa93a4cd8a81332e746ffd875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:40.220778 18191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce723a23a03497e907b58e878d9ba3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:40.220796 18322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce723a23a03497e907b58e878d9ba3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:40.221204 18060 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce723a23a03497e907b58e878d9ba3d (DEFAULT_TABLE table=test-table [id=dc8e874aa93a4cd8a81332e746ffd875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:40.221449 18060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce723a23a03497e907b58e878d9ba3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:40.224186 18402 tablet_bootstrap.cc:492] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d: Bootstrap starting.
I20260926 01:08:40.224751 18402 tablet_bootstrap.cc:654] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:40.224997 18402 log.cc:824] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:40.225301 18403 tablet_bootstrap.cc:492] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: Bootstrap starting.
I20260926 01:08:40.225847 18402 tablet_bootstrap.cc:492] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d: No bootstrap required, opened a new log
I20260926 01:08:40.225909 18403 tablet_bootstrap.cc:654] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:40.225926 18402 ts_tablet_manager.cc:1397] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:08:40.226213 18403 log.cc:824] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:40.226613 18404 tablet_bootstrap.cc:492] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: Bootstrap starting.
I20260926 01:08:40.226949 18403 tablet_bootstrap.cc:492] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: No bootstrap required, opened a new log
I20260926 01:08:40.227013 18403 ts_tablet_manager.cc:1397] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:08:40.227253 18404 tablet_bootstrap.cc:654] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:40.227563 18404 log.cc:824] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:40.228356 18404 tablet_bootstrap.cc:492] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: No bootstrap required, opened a new log
I20260926 01:08:40.228431 18404 ts_tablet_manager.cc:1397] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:08:40.228513 18402 raft_consensus.cc:348] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } } peers { permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 } } peers { permanent_uuid: "4159f55bc42f4aa39704f90411e9973d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41485 } }
I20260926 01:08:40.228683 18402 raft_consensus.cc:374] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:40.228722 18402 raft_consensus.cc:729] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4159f55bc42f4aa39704f90411e9973d, State: Initialized, Role: FOLLOWER
I20260926 01:08:40.228845 18402 consensus_queue.cc:260] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } } peers { permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 } } peers { permanent_uuid: "4159f55bc42f4aa39704f90411e9973d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41485 } }
I20260926 01:08:40.229120 18402 ts_tablet_manager.cc:1428] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 01:08:40.229254 18126 heartbeater.cc:503] Master 127.0.98.62:32929 was elected leader, sending a full tablet report...
I20260926 01:08:40.229645 18403 raft_consensus.cc:348] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } } peers { permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 } } peers { permanent_uuid: "4159f55bc42f4aa39704f90411e9973d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41485 } }
I20260926 01:08:40.229808 18403 raft_consensus.cc:374] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:40.229841 18403 raft_consensus.cc:729] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d64c053a904d6b9a6b5f80a7e03a01, State: Initialized, Role: FOLLOWER
I20260926 01:08:40.229950 18403 consensus_queue.cc:260] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } } peers { permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 } } peers { permanent_uuid: "4159f55bc42f4aa39704f90411e9973d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41485 } }
I20260926 01:08:40.230269 18403 ts_tablet_manager.cc:1428] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 01:08:40.230510 18388 heartbeater.cc:503] Master 127.0.98.62:32929 was elected leader, sending a full tablet report...
I20260926 01:08:40.230911 18404 raft_consensus.cc:348] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } } peers { permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 } } peers { permanent_uuid: "4159f55bc42f4aa39704f90411e9973d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41485 } }
I20260926 01:08:40.231048 18404 raft_consensus.cc:374] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:40.231079 18404 raft_consensus.cc:729] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412ff5721377485e9457cbf67471d890, State: Initialized, Role: FOLLOWER
I20260926 01:08:40.231194 18404 consensus_queue.cc:260] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } } peers { permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 } } peers { permanent_uuid: "4159f55bc42f4aa39704f90411e9973d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41485 } }
I20260926 01:08:40.231424 18404 ts_tablet_manager.cc:1428] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:40.231513 18257 heartbeater.cc:503] Master 127.0.98.62:32929 was elected leader, sending a full tablet report...
W20260926 01:08:40.303084 18258 tablet.cc:2369] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:08:40.398478 18127 tablet.cc:2369] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:08:40.423785 18408 raft_consensus.cc:482] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:08:40.424026 18408 raft_consensus.cc:504] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } } peers { permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 } } peers { permanent_uuid: "4159f55bc42f4aa39704f90411e9973d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41485 } }
I20260926 01:08:40.424448 18408 leader_election.cc:288] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412ff5721377485e9457cbf67471d890 (127.0.98.2:43787), a4d64c053a904d6b9a6b5f80a7e03a01 (127.0.98.3:42785)
I20260926 01:08:40.427585 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.424793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42785 (local address 127.0.98.1:48117)
0926 01:08:40.424883 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:40.424892 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:40.424946 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:40.425065 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:40.425066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:40.425068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:40.425148 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.425150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.425742 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.425743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.426791 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.426794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.427326 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.427327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.427388 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.427392 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:40.427437 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:08:40.427495 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":49,"thread_start_us":37,"threads_started":1}
I20260926 01:08:40.427572 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.424878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48117 (local address 127.0.98.3:42785)
0926 01:08:40.424931 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:40.424933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:40.424941 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:08:40.424999 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:40.425000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:40.425009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:40.425055 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:40.425205 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.425669 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.426885 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.427253 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.427317 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.427326 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:40.427459 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 01:08:40.427500 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:08:40.427937 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.424624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43787 (local address 127.0.98.1:42527)
0926 01:08:40.424743 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:40.424756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:40.424889 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:40.425061 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:40.425061 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:40.425064 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:40.425142 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.425143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.425749 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.425750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.427018 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:40.427021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.427616 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.427617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:40.427698 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.427702 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:40.427789 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 01:08:40.427824 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":112,"thread_start_us":63,"threads_started":1}
I20260926 01:08:40.427950 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.424714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42527 (local address 127.0.98.2:43787)
0926 01:08:40.424774 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:40.424776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:40.424879 (+ 103us) server_negotiation.cc:407] Connection header received
0926 01:08:40.424990 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:40.424991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:40.425003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:40.425052 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:40.425200 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.425678 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.427124 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.427526 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.427589 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.427598 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:40.427842 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 01:08:40.427895 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:08:40.427940 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce723a23a03497e907b58e878d9ba3d" candidate_uuid: "4159f55bc42f4aa39704f90411e9973d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" is_pre_election: true
I20260926 01:08:40.428172 18342 raft_consensus.cc:2454] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4159f55bc42f4aa39704f90411e9973d in term 0.
I20260926 01:08:40.428295 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce723a23a03497e907b58e878d9ba3d" candidate_uuid: "4159f55bc42f4aa39704f90411e9973d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412ff5721377485e9457cbf67471d890" is_pre_election: true
I20260926 01:08:40.428444 18211 raft_consensus.cc:2454] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4159f55bc42f4aa39704f90411e9973d in term 0.
I20260926 01:08:40.428431 18015 leader_election.cc:302] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4159f55bc42f4aa39704f90411e9973d, a4d64c053a904d6b9a6b5f80a7e03a01; no voters:
I20260926 01:08:40.428568 18408 raft_consensus.cc:2781] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:08:40.428609 18408 raft_consensus.cc:482] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:08:40.428625 18408 raft_consensus.cc:3037] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:40.429358 18408 raft_consensus.cc:504] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } } peers { permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 } } peers { permanent_uuid: "4159f55bc42f4aa39704f90411e9973d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41485 } }
I20260926 01:08:40.429481 18408 leader_election.cc:288] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [CANDIDATE]: Term 1 election: Requested vote from peers 412ff5721377485e9457cbf67471d890 (127.0.98.2:43787), a4d64c053a904d6b9a6b5f80a7e03a01 (127.0.98.3:42785)
I20260926 01:08:40.429706 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce723a23a03497e907b58e878d9ba3d" candidate_uuid: "4159f55bc42f4aa39704f90411e9973d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01"
I20260926 01:08:40.429703 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce723a23a03497e907b58e878d9ba3d" candidate_uuid: "4159f55bc42f4aa39704f90411e9973d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412ff5721377485e9457cbf67471d890"
I20260926 01:08:40.429785 18342 raft_consensus.cc:3037] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:40.429833 18211 raft_consensus.cc:3037] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:40.430639 18342 raft_consensus.cc:2454] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4159f55bc42f4aa39704f90411e9973d in term 1.
I20260926 01:08:40.430804 18211 raft_consensus.cc:2454] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4159f55bc42f4aa39704f90411e9973d in term 1.
I20260926 01:08:40.430814 18015 leader_election.cc:302] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4159f55bc42f4aa39704f90411e9973d, a4d64c053a904d6b9a6b5f80a7e03a01; no voters:
I20260926 01:08:40.430992 18408 raft_consensus.cc:2781] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:40.431146 18408 raft_consensus.cc:686] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [term 1 LEADER]: Becoming Leader. State: Replica: 4159f55bc42f4aa39704f90411e9973d, State: Running, Role: LEADER
I20260926 01:08:40.431239 18408 consensus_queue.cc:237] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } } peers { permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 } } peers { permanent_uuid: "4159f55bc42f4aa39704f90411e9973d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41485 } }
I20260926 01:08:40.432153 17937 catalog_manager.cc:5495] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4159f55bc42f4aa39704f90411e9973d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4159f55bc42f4aa39704f90411e9973d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4159f55bc42f4aa39704f90411e9973d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41485 } health_report { overall_health: HEALTHY } } }
W20260926 01:08:40.439760 18389 tablet.cc:2369] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:08:40.447713 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.445065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.0.98.1:41485)
0926 01:08:40.445146 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:40.445148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:40.445157 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:08:40.445193 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:40.445194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:40.445213 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:40.445284 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:08:40.445406 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.445894 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.446754 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:40.446955 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:40.447027 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:40.447264 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:08:40.447375 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:08:40.447463 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:40.447602 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 01:08:40.447626 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 01:08:40.450013 18342 raft_consensus.cc:1261] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01 [term 1 FOLLOWER]: Refusing update from remote peer 4159f55bc42f4aa39704f90411e9973d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:40.450023 18211 raft_consensus.cc:1261] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890 [term 1 FOLLOWER]: Refusing update from remote peer 4159f55bc42f4aa39704f90411e9973d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:40.450404 18408 consensus_queue.cc:1035] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d64c053a904d6b9a6b5f80a7e03a01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:40.450564 18413 consensus_queue.cc:1035] T 2ce723a23a03497e907b58e878d9ba3d P 4159f55bc42f4aa39704f90411e9973d [LEADER]: Connected to new peer: Peer: permanent_uuid: "412ff5721377485e9457cbf67471d890" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:40.454257 18418 mvcc.cc:205] Tried to move back new op lower bound from 7333416634159456256 to 7333416634087030784. Current Snapshot: MvccSnapshot[applied={T|T < 7333416634159456256}]
I20260926 01:08:40.456204 17937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:40.456336 17937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:40.457293 17937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fdbbc40ce1b4369867ea40c7672345f: Sending DeleteTablet for 3 replicas of tablet 2ce723a23a03497e907b58e878d9ba3d
I20260926 01:08:40.457762 18060 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce723a23a03497e907b58e878d9ba3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:40 UTC) from {username='slave'} at 127.0.0.1:56964
I20260926 01:08:40.457774 18191 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce723a23a03497e907b58e878d9ba3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:40 UTC) from {username='slave'} at 127.0.0.1:35748
I20260926 01:08:40.458101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 17998
I20260926 01:08:40.458076 18322 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce723a23a03497e907b58e878d9ba3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:40 UTC) from {username='slave'} at 127.0.0.1:37502
I20260926 01:08:40.458338 18427 tablet_replica.cc:330] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: stopping tablet replica
I20260926 01:08:40.459678 18427 raft_consensus.cc:2229] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:08:40.459860 18427 raft_consensus.cc:2258] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:40.460373 18427 ts_tablet_manager.cc:1905] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:40.462327 18427 ts_tablet_manager.cc:1918] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:40.462440 18427 log.cc:1196] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/2ce723a23a03497e907b58e878d9ba3d
I20260926 01:08:40.462574 18428 tablet_replica.cc:330] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: stopping tablet replica
I20260926 01:08:40.462677 18427 ts_tablet_manager.cc:1939] T 2ce723a23a03497e907b58e878d9ba3d P 412ff5721377485e9457cbf67471d890: Deleting consensus metadata
I20260926 01:08:40.462742 18428 raft_consensus.cc:2229] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:08:40.462880 18428 raft_consensus.cc:2258] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:40.463208 17924 catalog_manager.cc:4841] TS 412ff5721377485e9457cbf67471d890 (127.0.98.2:43787): tablet 2ce723a23a03497e907b58e878d9ba3d (table test-table [id=dc8e874aa93a4cd8a81332e746ffd875]) successfully deleted
I20260926 01:08:40.463454 18428 ts_tablet_manager.cc:1905] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:08:40.463930 17925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41485 (error 108)
I20260926 01:08:40.464457 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:40.464301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41485 (local address 127.0.0.1:56974)
0926 01:08:40.464363 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:40.464418 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 01:08:40.464455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 18129
I20260926 01:08:40.465317 18428 ts_tablet_manager.cc:1918] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:40.465384 18428 log.cc:1196] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/2ce723a23a03497e907b58e878d9ba3d
I20260926 01:08:40.465593 18428 ts_tablet_manager.cc:1939] T 2ce723a23a03497e907b58e878d9ba3d P a4d64c053a904d6b9a6b5f80a7e03a01: Deleting consensus metadata
I20260926 01:08:40.466320 17924 catalog_manager.cc:4841] TS a4d64c053a904d6b9a6b5f80a7e03a01 (127.0.98.3:42785): tablet 2ce723a23a03497e907b58e878d9ba3d (table test-table [id=dc8e874aa93a4cd8a81332e746ffd875]) successfully deleted
W20260926 01:08:40.466585 17925 catalog_manager.cc:4584] TS 4159f55bc42f4aa39704f90411e9973d (127.0.98.1:41485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ce723a23a03497e907b58e878d9ba3d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41485: connect: Connection refused (error 111)
I20260926 01:08:40.469686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 18260
I20260926 01:08:40.473791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 17907
2026-09-26T01:08:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (893 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:40 dist-test-slave-l73v krb5kdc[18432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:40 dist-test-slave-l73v krb5kdc[18432](info): set up 2 sockets
Sep 26 01:08:40 dist-test-slave-l73v krb5kdc[18432](info): commencing operation
krb5kdc: starting...
W20260926 01:08:41.434250 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:41 dist-test-slave-l73v krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:08:41.557113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:41.638365 18445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:41.638649 18445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:41.638674 18445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:41.639878 18445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:41.639920 18445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:41.639935 18445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:41.639945 18445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:41.639955 18445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:41.641477 18445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:41.641716 18445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:41.641912 18445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:41.646952 18453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:41.646955 18451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:41.647136 18445 server_base.cc:1034] running on GCE node
W20260926 01:08:41.646956 18450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:41.647550 18445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:41.647868 18445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:41.649042 18445 hybrid_clock.cc:648] HybridClock initialized: now 1790384921649008 us; error 54 us; skew 500 ppm
Sep 26 01:08:41 dist-test-slave-l73v krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:41.651695 18445 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:08:41.652194 18445 webserver.cc:460] Webserver started at http://127.0.98.62:39067/ using document root <none> and password file <none>
I20260926 01:08:41.652410 18445 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:41.652460 18445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:41.652585 18445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:41.653666 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "fffb5b39519b404b945434fe4f6afc76"
format_stamp: "Formatted at 2026-09-26 01:08:41 on dist-test-slave-l73v"
server_key: "7d1ee78e2b1bb478c0bcca964bece32f"
server_key_iv: "c39bdbd6fd2990a451956b4c06c2c2b7"
server_key_version: "encryptionkey@0"
I20260926 01:08:41.654017 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "fffb5b39519b404b945434fe4f6afc76"
format_stamp: "Formatted at 2026-09-26 01:08:41 on dist-test-slave-l73v"
server_key: "7d1ee78e2b1bb478c0bcca964bece32f"
server_key_iv: "c39bdbd6fd2990a451956b4c06c2c2b7"
server_key_version: "encryptionkey@0"
I20260926 01:08:41.655463 18445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:41.656309 18460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:41.656497 18445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:41.656584 18445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "fffb5b39519b404b945434fe4f6afc76"
format_stamp: "Formatted at 2026-09-26 01:08:41 on dist-test-slave-l73v"
server_key: "7d1ee78e2b1bb478c0bcca964bece32f"
server_key_iv: "c39bdbd6fd2990a451956b4c06c2c2b7"
server_key_version: "encryptionkey@0"
I20260926 01:08:41.656648 18445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:41.687634 18445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:41.689921 18445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:41.690065 18445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:41.694339 18445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36527
I20260926 01:08:41.694734 18512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36527 every 8 connection(s)
I20260926 01:08:41.694788 18445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:41.695461 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:41.697767 18513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76: Bootstrap starting.
I20260926 01:08:41.698369 18513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:41.698676 18513 log.cc:824] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:41.699361 18513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76: No bootstrap required, opened a new log
I20260926 01:08:41.701006 18513 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffb5b39519b404b945434fe4f6afc76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260926 01:08:41.701161 18513 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:41.701182 18513 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffb5b39519b404b945434fe4f6afc76, State: Initialized, Role: FOLLOWER
I20260926 01:08:41.701318 18513 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffb5b39519b404b945434fe4f6afc76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260926 01:08:41.701386 18513 raft_consensus.cc:388] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:41.701409 18513 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:41.701457 18513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:41.702026 18513 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffb5b39519b404b945434fe4f6afc76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260926 01:08:41.702116 18513 leader_election.cc:302] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fffb5b39519b404b945434fe4f6afc76; no voters:
I20260926 01:08:41.702301 18513 leader_election.cc:288] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:41.702327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 18445
I20260926 01:08:41.702355 18516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:41.702454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:41.702613 18513 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:41.702651 392 external_mini_cluster.cc:1428] Setting key 5734cda401319e52ea96e0bc61c6c905
I20260926 01:08:41.702739 18516 raft_consensus.cc:686] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [term 1 LEADER]: Becoming Leader. State: Replica: fffb5b39519b404b945434fe4f6afc76, State: Running, Role: LEADER
I20260926 01:08:41.702843 18516 consensus_queue.cc:237] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffb5b39519b404b945434fe4f6afc76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260926 01:08:41.703156 18516 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fffb5b39519b404b945434fe4f6afc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffb5b39519b404b945434fe4f6afc76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } }
I20260926 01:08:41.703236 18516 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:41.703357 18516 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fffb5b39519b404b945434fe4f6afc76. Latest consensus state: current_term: 1 leader_uuid: "fffb5b39519b404b945434fe4f6afc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffb5b39519b404b945434fe4f6afc76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } }
I20260926 01:08:41.703413 18516 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76 [sys.catalog]: This master's current role is: LEADER
W20260926 01:08:41.704454 18533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:08:41.704537 18533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:08:41.704587 18526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:41.704751 18526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:41.706888 18526 catalog_manager.cc:1347] Generated new cluster ID: 0f4673e513c34fafbedbbdf7bf1e5df3
I20260926 01:08:41.706952 18526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:08:41 dist-test-slave-l73v krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:41.714322 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:41.703297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.98.62:36527)
0926 01:08:41.703497 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:41.703507 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:41.703538 (+ 31us) server_negotiation.cc:407] Connection header received
0926 01:08:41.703656 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:41.703664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:41.703737 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:41.704409 (+ 672us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:41.704644 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:41.705902 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:41.706843 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:41.707419 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:41.707520 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:41.710304 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:41.710311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:41.710311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:41.710327 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:41.712353 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:41.712851 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:41.712854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:41.712855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:41.712979 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:41.713196 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:41.713196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:41.713197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:41.713634 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:41.713816 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:41.713893 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 01:08:41.714048 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260926 01:08:41.730846 18526 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:41.731684 18526 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:41.743953 18526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76: Generated new TSK 0
I20260926 01:08:41.744176 18526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:08:41.779325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:41.865036 18539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:41.865260 18539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:41.865285 18539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:41.866606 18539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:41.866663 18539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:41.866693 18539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:08:41.868384 18539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:41.868650 18539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:41.868886 18539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:41.874421 18545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:41.874533 18544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:41.874441 18547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:41.874676 18539 server_base.cc:1034] running on GCE node
I20260926 01:08:41.874902 18539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:41.875169 18539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:41.876349 18539 hybrid_clock.cc:648] HybridClock initialized: now 1790384921876312 us; error 51 us; skew 500 ppm
Sep 26 01:08:41 dist-test-slave-l73v krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:41.884676 18539 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:08:41.885252 18539 webserver.cc:460] Webserver started at http://127.0.98.1:36063/ using document root <none> and password file <none>
I20260926 01:08:41.885500 18539 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:41.885563 18539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:41.885704 18539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:41.886909 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "f1d1a6f845534be38426a23986ee1d6e"
format_stamp: "Formatted at 2026-09-26 01:08:41 on dist-test-slave-l73v"
server_key: "38c5c3409f5f4c70beed25a6f515a8d0"
server_key_iv: "7444f3fc09c50353f57cc0aa0e8c6c23"
server_key_version: "encryptionkey@0"
I20260926 01:08:41.887499 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f1d1a6f845534be38426a23986ee1d6e"
format_stamp: "Formatted at 2026-09-26 01:08:41 on dist-test-slave-l73v"
server_key: "38c5c3409f5f4c70beed25a6f515a8d0"
server_key_iv: "7444f3fc09c50353f57cc0aa0e8c6c23"
server_key_version: "encryptionkey@0"
I20260926 01:08:41.889181 18539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:08:41.890427 18554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:41.890779 18539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:41.891024 18539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "f1d1a6f845534be38426a23986ee1d6e"
format_stamp: "Formatted at 2026-09-26 01:08:41 on dist-test-slave-l73v"
server_key: "38c5c3409f5f4c70beed25a6f515a8d0"
server_key_iv: "7444f3fc09c50353f57cc0aa0e8c6c23"
server_key_version: "encryptionkey@0"
I20260926 01:08:41.891121 18539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:41.911088 18539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:41.914235 18539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:41.914543 18539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:41.914812 18539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:41.915133 18539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:41.915169 18539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:41.915200 18539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:41.915226 18539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:41.921795 18539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46603
I20260926 01:08:41.921847 18667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46603 every 8 connection(s)
I20260926 01:08:41.922200 18539 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:08:41.924816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 18539
I20260926 01:08:41.924937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:08:41.925146 392 external_mini_cluster.cc:1428] Setting key 12efe96ab575665a94c70f8cdf3f82fa
Sep 26 01:08:41 dist-test-slave-l73v krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:08:41.934615 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:41.923020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39277 (local address 127.0.98.62:36527)
0926 01:08:41.923105 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:41.923107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:41.923537 (+ 430us) server_negotiation.cc:407] Connection header received
0926 01:08:41.925020 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:41.925022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:41.925041 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:41.925130 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:41.925969 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:41.926549 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:41.927626 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:41.927844 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:41.927903 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:41.930607 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:41.930609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:41.930610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:41.930624 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:41.932311 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:41.932854 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:41.932856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:41.932856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:41.932928 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:41.933232 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:41.933233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:41.933234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:41.933388 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:41.933469 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:41.934108 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 01:08:41.934280 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 01:08:41.934768 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:41.922989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.1:39277)
0926 01:08:41.923302 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:41.923377 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:41.924778 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:41.925230 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:41.925236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:41.925545 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:41.925850 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:41.925857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:41.926648 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:41.926649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:41.927520 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:41.927523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:41.927924 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:41.927926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:41.928052 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:41.928608 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:41.928617 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:41.930489 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:41.932428 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:41.932438 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:41.932441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:41.932748 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:41.933024 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:41.933025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:41.933025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:41.933146 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:41.933564 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:41.933580 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:41.933819 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 01:08:41.934019 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 01:08:41.934280 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":93,"threads_started":1}
I20260926 01:08:41.935479 18668 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36527
I20260926 01:08:41.935633 18668 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:41.935819 18668 heartbeater.cc:511] Master 127.0.98.62:36527 requested a full tablet report, sending...
I20260926 01:08:41.936390 18477 ts_manager.cc:194] Registered new tserver with Master: f1d1a6f845534be38426a23986ee1d6e (127.0.98.1:46603)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 01:08:41.937153 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:08:41.954842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:42.038643 18673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:42.038890 18673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:42.038929 18673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:42.040904 18673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:42.040990 18673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:42.041044 18673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:08:42.043521 18673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:42.043843 18673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:42.044080 18673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:42.049244 18681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:42.049253 18678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.049270 18673 server_base.cc:1034] running on GCE node
W20260926 01:08:42.049348 18679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.049645 18673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:42.049904 18673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:42.051092 18673 hybrid_clock.cc:648] HybridClock initialized: now 1790384922051065 us; error 36 us; skew 500 ppm
Sep 26 01:08:42 dist-test-slave-l73v krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:42.053678 18673 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:08:42.054176 18673 webserver.cc:460] Webserver started at http://127.0.98.2:43995/ using document root <none> and password file <none>
I20260926 01:08:42.054425 18673 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:42.054479 18673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:42.054606 18673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:42.055697 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "45d3122de76f4103817363a98e241dd7"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "5f73c11cd5d1c8be3964654c969eeed1"
server_key_iv: "cffaff0bb666c4d263d028f1b96e3ba9"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.056034 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "45d3122de76f4103817363a98e241dd7"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "5f73c11cd5d1c8be3964654c969eeed1"
server_key_iv: "cffaff0bb666c4d263d028f1b96e3ba9"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.057340 18673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:42.058156 18688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.058329 18673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:42.058440 18673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "45d3122de76f4103817363a98e241dd7"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "5f73c11cd5d1c8be3964654c969eeed1"
server_key_iv: "cffaff0bb666c4d263d028f1b96e3ba9"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.058513 18673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:42.084517 18673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:42.086824 18673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:42.086967 18673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:42.087209 18673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:42.087503 18673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:42.087538 18673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.087566 18673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:42.087584 18673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.093364 18673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46819
I20260926 01:08:42.093410 18801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46819 every 8 connection(s)
I20260926 01:08:42.093735 18673 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:08:42 dist-test-slave-l73v krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:42.100282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 18673
I20260926 01:08:42.100369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:08:42.100596 392 external_mini_cluster.cc:1428] Setting key 7559eb36fffbe294134e4f66bcb4c4fb
I20260926 01:08:42.103672 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.094471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50061 (local address 127.0.98.62:36527)
0926 01:08:42.094550 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.094552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.094790 (+ 238us) server_negotiation.cc:407] Connection header received
0926 01:08:42.095814 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.095816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.095828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.095882 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:42.096517 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.097023 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.098033 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.098247 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.098307 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.100491 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:42.100493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:42.100493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:42.100501 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:42.102080 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:42.102515 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:42.102516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:42.102516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:42.102588 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:42.102939 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:42.102940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:42.102941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:42.103124 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:42.103186 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.103459 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.103570 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 01:08:42.103785 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.094472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.2:50061)
0926 01:08:42.094669 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.094707 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.095709 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.095958 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.095962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.096230 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:42.096426 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.096431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.097114 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.097115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.097946 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.097949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.098326 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.098327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.098500 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.099013 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:42.099022 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:42.100399 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:42.102156 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:42.102164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:42.102166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:42.102449 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:42.102689 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:42.102691 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:42.102691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:42.102863 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:42.103176 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:42.103182 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:42.103339 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.103419 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.103583 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
I20260926 01:08:42.104242 18802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36527
I20260926 01:08:42.104355 18802 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:42.104547 18802 heartbeater.cc:511] Master 127.0.98.62:36527 requested a full tablet report, sending...
I20260926 01:08:42.104955 18477 ts_manager.cc:194] Registered new tserver with Master: 45d3122de76f4103817363a98e241dd7 (127.0.98.2:46819)
I20260926 01:08:42.105463 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50061
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:08:42.131282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:42.213519 18807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:42.213768 18807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:42.213801 18807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:42.215171 18807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:42.215234 18807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:42.215276 18807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:08:42.217043 18807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:42.217322 18807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:42.217559 18807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:42.222784 18815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:42.222785 18812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.222941 18807 server_base.cc:1034] running on GCE node
W20260926 01:08:42.222788 18813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.223351 18807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:42.223733 18807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:42.224917 18807 hybrid_clock.cc:648] HybridClock initialized: now 1790384922224894 us; error 38 us; skew 500 ppm
Sep 26 01:08:42 dist-test-slave-l73v krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:42.227885 18807 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:08:42.228365 18807 webserver.cc:460] Webserver started at http://127.0.98.3:34645/ using document root <none> and password file <none>
I20260926 01:08:42.228571 18807 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:42.228619 18807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:42.228744 18807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:42.229811 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "ffb841ec4105492bb23f938866f0de91"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "5a67c13b49c54ba15995f7e8efd0a7de"
server_key_iv: "6ce1f21fed33c4a8e790ddb5312fa4bd"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.230145 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ffb841ec4105492bb23f938866f0de91"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "5a67c13b49c54ba15995f7e8efd0a7de"
server_key_iv: "6ce1f21fed33c4a8e790ddb5312fa4bd"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.231496 18807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 01:08:42.232266 18822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.232434 18807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:08:42.232515 18807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "ffb841ec4105492bb23f938866f0de91"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "5a67c13b49c54ba15995f7e8efd0a7de"
server_key_iv: "6ce1f21fed33c4a8e790ddb5312fa4bd"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.232606 18807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:42.251055 18807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:42.253190 18807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:42.253331 18807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:42.253549 18807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:42.253835 18807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:42.253868 18807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.253896 18807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:42.253921 18807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.259519 18807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41771
I20260926 01:08:42.259629 18935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41771 every 8 connection(s)
I20260926 01:08:42.259893 18807 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 01:08:42 dist-test-slave-l73v krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:42.266417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 18807
I20260926 01:08:42.266512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:08:42.266714 392 external_mini_cluster.cc:1428] Setting key 704deb1163ef618b73bfddc2c5fa8df4
I20260926 01:08:42.269173 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.260599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33649 (local address 127.0.98.62:36527)
0926 01:08:42.260677 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.260679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.260813 (+ 134us) server_negotiation.cc:407] Connection header received
0926 01:08:42.261806 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.261807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.261818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.261873 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:42.262515 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.263014 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.263979 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.264166 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.264221 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.266301 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:42.266303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:42.266303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:42.266311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:42.267815 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:42.268161 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:42.268162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:42.268162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:42.268208 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:42.268555 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:42.268556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:42.268556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:42.268680 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:42.268741 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.269027 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.269105 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 01:08:42.269280 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.260556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.3:33649)
0926 01:08:42.260699 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.260736 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.261701 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.261931 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.261934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.262200 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:42.262420 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.262426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.263086 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.263087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.263898 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.263900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.264250 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.264251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.264353 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.264831 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:42.264839 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:42.266209 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:42.267885 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:42.267891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:42.267893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:42.268103 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:42.268282 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:42.268283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:42.268284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:42.268427 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:42.268741 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:42.268746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:42.268871 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.268965 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.269098 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260926 01:08:42.269840 18936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36527
I20260926 01:08:42.269990 18936 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:42.270155 18936 heartbeater.cc:511] Master 127.0.98.62:36527 requested a full tablet report, sending...
I20260926 01:08:42.270557 18477 ts_manager.cc:194] Registered new tserver with Master: ffb841ec4105492bb23f938866f0de91 (127.0.98.3:41771)
I20260926 01:08:42.270968 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33649
I20260926 01:08:42.278565 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:08:42.286008 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.279877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.0.98.62:36527)
0926 01:08:42.279928 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.279930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.279982 (+ 52us) server_negotiation.cc:407] Connection header received
0926 01:08:42.280053 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.280053 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.280062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.280103 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:42.280644 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.281158 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.281925 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.282136 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.282195 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.283553 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:42.283555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:42.283555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:42.283563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:42.284765 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:42.285310 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:42.285311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:42.285312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:42.285369 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:42.285568 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:42.285569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:42.285569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:42.285701 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:42.285792 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.285877 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.285930 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:08:42.287650 18477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:08:42.289281 18477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:08:42.295826 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.291914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.0.98.1:46603)
0926 01:08:42.292080 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.292084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.292093 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:08:42.292129 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.292131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.292142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.292212 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:42.292385 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.293454 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.294813 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.295403 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.295469 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.295530 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.295688 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.295737 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260926 01:08:42.295874 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.291773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.0.98.2:46819)
0926 01:08:42.292048 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.292053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.292063 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:08:42.292105 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.292107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.292124 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.292199 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:42.292388 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.293258 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.294780 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.295517 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.295599 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.295652 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.295748 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.295790 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":74,"threads_started":1}
I20260926 01:08:42.295853 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.291637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46819 (local address 127.0.0.1:39166)
0926 01:08:42.291789 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.291829 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.291894 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.292214 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.292217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.292219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:42.292303 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.292306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.293351 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.293352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.294681 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.294683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.295602 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.295603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.295670 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.295674 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.295723 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.295785 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":3,"thread_start_us":46,"threads_started":1}
I20260926 01:08:42.296015 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.291877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.0.98.3:41771)
0926 01:08:42.292341 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.292345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.292354 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:08:42.292388 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.292390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.292403 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.292485 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:42.292785 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.293783 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.294882 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.295539 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.295597 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.295652 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.295892 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.295929 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":69,"threads_started":1}
I20260926 01:08:42.296142 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.291653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41771 (local address 127.0.0.1:59106)
0926 01:08:42.292078 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.292088 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.292128 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.292674 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.292675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.292677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:42.292710 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.292711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.293868 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.293870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.294811 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.294813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.295685 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.295686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.295748 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.295753 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.295797 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.295840 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":98,"threads_started":1}
I20260926 01:08:42.296444 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.291637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46603 (local address 127.0.0.1:60152)
0926 01:08:42.291941 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.291952 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.292014 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.292219 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.292220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.292223 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:42.292301 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.292303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.293556 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.293557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.294681 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.294683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.295478 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.295479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.295564 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.295580 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.296343 (+ 763us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.296394 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260926 01:08:42.296913 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa311bf49bb4999b24258af4f064866 (DEFAULT_TABLE table=test-table [id=1fc7c4c254ea4614bce4274abe3122e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:42.296941 18870 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa311bf49bb4999b24258af4f064866 (DEFAULT_TABLE table=test-table [id=1fc7c4c254ea4614bce4274abe3122e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:42.297107 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa311bf49bb4999b24258af4f064866 (DEFAULT_TABLE table=test-table [id=1fc7c4c254ea4614bce4274abe3122e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:42.297267 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa311bf49bb4999b24258af4f064866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:42.297276 18870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa311bf49bb4999b24258af4f064866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:42.297369 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa311bf49bb4999b24258af4f064866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:42.299923 18953 tablet_bootstrap.cc:492] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: Bootstrap starting.
I20260926 01:08:42.300439 18954 tablet_bootstrap.cc:492] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e: Bootstrap starting.
I20260926 01:08:42.300566 18953 tablet_bootstrap.cc:654] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:42.301074 18954 tablet_bootstrap.cc:654] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:42.301054 18955 tablet_bootstrap.cc:492] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: Bootstrap starting.
I20260926 01:08:42.301148 18953 log.cc:824] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:42.301585 18954 log.cc:824] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:42.301702 18955 tablet_bootstrap.cc:654] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:42.302031 18953 tablet_bootstrap.cc:492] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: No bootstrap required, opened a new log
I20260926 01:08:42.302094 18953 ts_tablet_manager.cc:1397] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:08:42.302160 18955 log.cc:824] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:42.302868 18954 tablet_bootstrap.cc:492] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e: No bootstrap required, opened a new log
I20260926 01:08:42.302912 18955 tablet_bootstrap.cc:492] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: No bootstrap required, opened a new log
I20260926 01:08:42.302934 18954 ts_tablet_manager.cc:1397] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 01:08:42.302973 18955 ts_tablet_manager.cc:1397] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:08:42.304142 18953 raft_consensus.cc:348] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } } peers { permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "ffb841ec4105492bb23f938866f0de91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41771 } }
I20260926 01:08:42.304325 18953 raft_consensus.cc:374] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:42.304373 18953 raft_consensus.cc:729] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb841ec4105492bb23f938866f0de91, State: Initialized, Role: FOLLOWER
I20260926 01:08:42.304399 18954 raft_consensus.cc:348] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } } peers { permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "ffb841ec4105492bb23f938866f0de91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41771 } }
I20260926 01:08:42.304497 18953 consensus_queue.cc:260] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } } peers { permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "ffb841ec4105492bb23f938866f0de91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41771 } }
I20260926 01:08:42.304556 18954 raft_consensus.cc:374] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:42.304584 18954 raft_consensus.cc:729] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d1a6f845534be38426a23986ee1d6e, State: Initialized, Role: FOLLOWER
I20260926 01:08:42.304667 18954 consensus_queue.cc:260] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } } peers { permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "ffb841ec4105492bb23f938866f0de91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41771 } }
I20260926 01:08:42.304767 18953 ts_tablet_manager.cc:1428] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:42.304880 18936 heartbeater.cc:503] Master 127.0.98.62:36527 was elected leader, sending a full tablet report...
I20260926 01:08:42.304919 18954 ts_tablet_manager.cc:1428] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:08:42.304930 18668 heartbeater.cc:503] Master 127.0.98.62:36527 was elected leader, sending a full tablet report...
I20260926 01:08:42.304984 18955 raft_consensus.cc:348] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } } peers { permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "ffb841ec4105492bb23f938866f0de91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41771 } }
I20260926 01:08:42.305131 18955 raft_consensus.cc:374] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:42.305163 18955 raft_consensus.cc:729] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d3122de76f4103817363a98e241dd7, State: Initialized, Role: FOLLOWER
I20260926 01:08:42.305263 18955 consensus_queue.cc:260] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } } peers { permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "ffb841ec4105492bb23f938866f0de91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41771 } }
I20260926 01:08:42.305523 18955 ts_tablet_manager.cc:1428] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:42.305526 18802 heartbeater.cc:503] Master 127.0.98.62:36527 was elected leader, sending a full tablet report...
I20260926 01:08:42.307798 18959 raft_consensus.cc:482] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:08:42.307884 18959 raft_consensus.cc:504] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } } peers { permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "ffb841ec4105492bb23f938866f0de91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41771 } }
I20260926 01:08:42.308187 18959 leader_election.cc:288] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45d3122de76f4103817363a98e241dd7 (127.0.98.2:46819), f1d1a6f845534be38426a23986ee1d6e (127.0.98.1:46603)
I20260926 01:08:42.311044 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.308447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54293 (local address 127.0.98.1:46603)
0926 01:08:42.308489 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.308491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.308534 (+ 43us) server_negotiation.cc:407] Connection header received
0926 01:08:42.308594 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.308595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.308605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.308661 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:42.308787 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.309284 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.310201 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.310724 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.310778 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.310790 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.310941 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.310983 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:08:42.311067 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.308354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46603 (local address 127.0.98.3:54293)
0926 01:08:42.308490 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.308500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.308549 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.308665 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.308666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.308667 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:42.308732 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.308734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.309360 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.309361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.310102 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.310104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.310806 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.310807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.310883 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.310886 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.310929 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.310971 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 01:08:42.311079 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.308565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40809 (local address 127.0.98.2:46819)
0926 01:08:42.308607 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.308609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.308616 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:08:42.308677 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.308678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.308687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.308738 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:42.308908 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.309340 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.310278 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.310774 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.310832 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.310841 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.310980 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.311015 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:08:42.311389 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa311bf49bb4999b24258af4f064866" candidate_uuid: "ffb841ec4105492bb23f938866f0de91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d1a6f845534be38426a23986ee1d6e" is_pre_election: true
I20260926 01:08:42.311496 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.308364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46819 (local address 127.0.98.3:40809)
0926 01:08:42.308566 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.308576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.308630 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.308742 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.308743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.308744 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:42.308810 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.308811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.309411 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.309412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.310145 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.310147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.310841 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.310842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.310922 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.310925 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.310970 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.311010 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":39,"thread_start_us":104,"threads_started":1}
I20260926 01:08:42.311525 18622 raft_consensus.cc:2454] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb841ec4105492bb23f938866f0de91 in term 0.
I20260926 01:08:42.311736 18826 leader_election.cc:302] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1d1a6f845534be38426a23986ee1d6e, ffb841ec4105492bb23f938866f0de91; no voters:
I20260926 01:08:42.311789 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa311bf49bb4999b24258af4f064866" candidate_uuid: "ffb841ec4105492bb23f938866f0de91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d3122de76f4103817363a98e241dd7" is_pre_election: true
I20260926 01:08:42.311862 18959 raft_consensus.cc:2781] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:08:42.311914 18959 raft_consensus.cc:482] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:08:42.311928 18959 raft_consensus.cc:3037] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:42.311923 18756 raft_consensus.cc:2454] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb841ec4105492bb23f938866f0de91 in term 0.
I20260926 01:08:42.312597 18959 raft_consensus.cc:504] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } } peers { permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "ffb841ec4105492bb23f938866f0de91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41771 } }
I20260926 01:08:42.312707 18959 leader_election.cc:288] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [CANDIDATE]: Term 1 election: Requested vote from peers 45d3122de76f4103817363a98e241dd7 (127.0.98.2:46819), f1d1a6f845534be38426a23986ee1d6e (127.0.98.1:46603)
I20260926 01:08:42.312872 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa311bf49bb4999b24258af4f064866" candidate_uuid: "ffb841ec4105492bb23f938866f0de91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d1a6f845534be38426a23986ee1d6e"
I20260926 01:08:42.312836 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa311bf49bb4999b24258af4f064866" candidate_uuid: "ffb841ec4105492bb23f938866f0de91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d3122de76f4103817363a98e241dd7"
I20260926 01:08:42.312955 18622 raft_consensus.cc:3037] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:42.312955 18756 raft_consensus.cc:3037] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:42.313794 18622 raft_consensus.cc:2454] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb841ec4105492bb23f938866f0de91 in term 1.
I20260926 01:08:42.313794 18756 raft_consensus.cc:2454] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb841ec4105492bb23f938866f0de91 in term 1.
I20260926 01:08:42.313964 18825 leader_election.cc:302] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d3122de76f4103817363a98e241dd7, ffb841ec4105492bb23f938866f0de91; no voters:
I20260926 01:08:42.314055 18959 raft_consensus.cc:2781] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:42.314208 18959 raft_consensus.cc:686] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 1 LEADER]: Becoming Leader. State: Replica: ffb841ec4105492bb23f938866f0de91, State: Running, Role: LEADER
I20260926 01:08:42.314301 18959 consensus_queue.cc:237] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } } peers { permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "ffb841ec4105492bb23f938866f0de91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41771 } }
I20260926 01:08:42.315156 18477 catalog_manager.cc:5495] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffb841ec4105492bb23f938866f0de91 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ffb841ec4105492bb23f938866f0de91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffb841ec4105492bb23f938866f0de91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41771 } health_report { overall_health: HEALTHY } } }
I20260926 01:08:42.326808 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.324457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.0.98.3:41771)
0926 01:08:42.324505 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.324506 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.324515 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:08:42.324569 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.324570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.324586 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.324643 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:08:42.324746 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.325207 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.325923 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.326136 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.326199 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.326452 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:08:42.326562 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:08:42.326644 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.326710 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.326718 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:08:42.328637 18622 raft_consensus.cc:1261] T 1aa311bf49bb4999b24258af4f064866 P f1d1a6f845534be38426a23986ee1d6e [term 1 FOLLOWER]: Refusing update from remote peer ffb841ec4105492bb23f938866f0de91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:42.328642 18756 raft_consensus.cc:1261] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7 [term 1 FOLLOWER]: Refusing update from remote peer ffb841ec4105492bb23f938866f0de91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:42.328979 18963 consensus_queue.cc:1035] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1d1a6f845534be38426a23986ee1d6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:42.329124 18959 consensus_queue.cc:1035] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d3122de76f4103817363a98e241dd7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:42.332269 18968 mvcc.cc:205] Tried to move back new op lower bound from 7333416641855627264 to 7333416641800433664. Current Snapshot: MvccSnapshot[applied={T|T < 7333416641855627264}]
I20260926 01:08:42.332340 18966 mvcc.cc:205] Tried to move back new op lower bound from 7333416641855627264 to 7333416641800433664. Current Snapshot: MvccSnapshot[applied={T|T < 7333416641855627264 or (T in {7333416641855627264})}]
I20260926 01:08:42.332998 18967 mvcc.cc:205] Tried to move back new op lower bound from 7333416641855627264 to 7333416641800433664. Current Snapshot: MvccSnapshot[applied={T|T < 7333416641855627264}]
I20260926 01:08:42.334190 18477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:42.334319 18477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:42.335117 18477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fffb5b39519b404b945434fe4f6afc76: Sending DeleteTablet for 3 replicas of tablet 1aa311bf49bb4999b24258af4f064866
I20260926 01:08:42.335457 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aa311bf49bb4999b24258af4f064866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39166
I20260926 01:08:42.335610 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aa311bf49bb4999b24258af4f064866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60152
I20260926 01:08:42.335678 18870 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aa311bf49bb4999b24258af4f064866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59106
I20260926 01:08:42.335907 18977 tablet_replica.cc:330] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: stopping tablet replica
I20260926 01:08:42.335901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 18539
I20260926 01:08:42.336428 18977 raft_consensus.cc:2229] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:08:42.336596 18977 raft_consensus.cc:2258] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:42.337072 18977 ts_tablet_manager.cc:1905] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:42.337922 18975 tablet_replica.cc:330] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: stopping tablet replica
I20260926 01:08:42.338074 18975 raft_consensus.cc:2229] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:08:42.338231 18975 raft_consensus.cc:2258] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:42.338857 18975 ts_tablet_manager.cc:1905] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:42.338956 18977 ts_tablet_manager.cc:1918] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:42.339037 18977 log.cc:1196] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/1aa311bf49bb4999b24258af4f064866
I20260926 01:08:42.339365 18977 ts_tablet_manager.cc:1939] T 1aa311bf49bb4999b24258af4f064866 P ffb841ec4105492bb23f938866f0de91: Deleting consensus metadata
I20260926 01:08:42.339758 18462 catalog_manager.cc:4841] TS ffb841ec4105492bb23f938866f0de91 (127.0.98.3:41771): tablet 1aa311bf49bb4999b24258af4f064866 (table test-table [id=1fc7c4c254ea4614bce4274abe3122e9]) successfully deleted
I20260926 01:08:42.340718 18975 ts_tablet_manager.cc:1918] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:42.340797 18975 log.cc:1196] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/1aa311bf49bb4999b24258af4f064866
I20260926 01:08:42.341069 18975 ts_tablet_manager.cc:1939] T 1aa311bf49bb4999b24258af4f064866 P 45d3122de76f4103817363a98e241dd7: Deleting consensus metadata
I20260926 01:08:42.341516 18463 catalog_manager.cc:4841] TS 45d3122de76f4103817363a98e241dd7 (127.0.98.2:46819): tablet 1aa311bf49bb4999b24258af4f064866 (table test-table [id=1fc7c4c254ea4614bce4274abe3122e9]) successfully deleted
W20260926 01:08:42.345250 18464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46603 (error 108)
I20260926 01:08:42.345377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 18673
I20260926 01:08:42.347939 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.347806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46603 (local address 127.0.0.1:60160)
0926 01:08:42.347859 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.347904 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 01:08:42.348188 18464 catalog_manager.cc:4584] TS f1d1a6f845534be38426a23986ee1d6e (127.0.98.1:46603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1aa311bf49bb4999b24258af4f064866: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46603: connect: Connection refused (error 111)
I20260926 01:08:42.349939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 18807
I20260926 01:08:42.354044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 18445
2026-09-26T01:08:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1880 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T01:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:42Z Disabled control of system clock
I20260926 01:08:42.380183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34255
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:08:42.460296 18982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:42.460561 18982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:42.460582 18982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:42.462020 18982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:42.462081 18982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:42.462092 18982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:42.462105 18982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:42.462116 18982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:42.463636 18982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:42.463886 18982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:42.464187 18982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:42.469235 18990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.469259 18982 server_base.cc:1034] running on GCE node
W20260926 01:08:42.469226 18988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:42.469206 18987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.469668 18982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:42.469934 18982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:42.471096 18982 hybrid_clock.cc:648] HybridClock initialized: now 1790384922471067 us; error 46 us; skew 500 ppm
I20260926 01:08:42.472414 18982 webserver.cc:460] Webserver started at http://127.0.98.62:36399/ using document root <none> and password file <none>
I20260926 01:08:42.472651 18982 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:42.472697 18982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:42.472860 18982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:42.473960 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "e42615d3a5494eb1a8ebb569f3b66ebd"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "df573658008b22cf32977d76103e424d"
server_key_iv: "31f1ffaef2666ba80d0673281a341609"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.474298 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "e42615d3a5494eb1a8ebb569f3b66ebd"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "df573658008b22cf32977d76103e424d"
server_key_iv: "31f1ffaef2666ba80d0673281a341609"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.475620 18982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:08:42.476400 18996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.476541 18982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:08:42.476620 18982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "e42615d3a5494eb1a8ebb569f3b66ebd"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "df573658008b22cf32977d76103e424d"
server_key_iv: "31f1ffaef2666ba80d0673281a341609"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.476691 18982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:42.500969 18982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:42.501313 18982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:42.501444 18982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:42.505877 18982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34255
I20260926 01:08:42.505940 19048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34255 every 8 connection(s)
I20260926 01:08:42.506285 18982 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:42.507323 19049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:42.509728 19049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd: Bootstrap starting.
I20260926 01:08:42.510592 19049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:42.510962 19049 log.cc:824] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:42.511729 19049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd: No bootstrap required, opened a new log
I20260926 01:08:42.513468 19049 raft_consensus.cc:348] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42615d3a5494eb1a8ebb569f3b66ebd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } }
I20260926 01:08:42.513629 19049 raft_consensus.cc:374] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:42.513651 19049 raft_consensus.cc:729] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42615d3a5494eb1a8ebb569f3b66ebd, State: Initialized, Role: FOLLOWER
I20260926 01:08:42.513794 19049 consensus_queue.cc:260] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42615d3a5494eb1a8ebb569f3b66ebd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } }
I20260926 01:08:42.513870 19049 raft_consensus.cc:388] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:42.513893 19049 raft_consensus.cc:482] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:42.513923 19049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:42.514559 19049 raft_consensus.cc:504] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42615d3a5494eb1a8ebb569f3b66ebd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } }
I20260926 01:08:42.514647 19049 leader_election.cc:302] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e42615d3a5494eb1a8ebb569f3b66ebd; no voters:
I20260926 01:08:42.514829 19049 leader_election.cc:288] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:42.514880 19052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:42.514922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 18982
I20260926 01:08:42.515005 19052 raft_consensus.cc:686] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [term 1 LEADER]: Becoming Leader. State: Replica: e42615d3a5494eb1a8ebb569f3b66ebd, State: Running, Role: LEADER
I20260926 01:08:42.515034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:42.515111 19049 sys_catalog.cc:564] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:42.515233 392 external_mini_cluster.cc:1428] Setting key f57d1c722aa108e518bd575c3a146867
I20260926 01:08:42.515108 19052 consensus_queue.cc:237] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42615d3a5494eb1a8ebb569f3b66ebd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } }
I20260926 01:08:42.515825 19053 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e42615d3a5494eb1a8ebb569f3b66ebd. Latest consensus state: current_term: 1 leader_uuid: "e42615d3a5494eb1a8ebb569f3b66ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42615d3a5494eb1a8ebb569f3b66ebd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } } }
I20260926 01:08:42.515895 19053 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:42.516448 19054 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e42615d3a5494eb1a8ebb569f3b66ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42615d3a5494eb1a8ebb569f3b66ebd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } } }
I20260926 01:08:42.516510 19062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:42.516664 19054 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:42.517266 19062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:42.519011 19062 catalog_manager.cc:1347] Generated new cluster ID: 4af8664be175470c8ab7b21a8515e1ff
I20260926 01:08:42.519073 19062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:42.521278 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.515760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.0.98.62:34255)
0926 01:08:42.516515 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.516525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.516564 (+ 39us) server_negotiation.cc:407] Connection header received
0926 01:08:42.516709 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.516716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.516789 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.516977 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:42.517198 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.518435 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.519479 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.519913 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.520024 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.520649 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:42.520653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:42.520653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:42.520818 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:42.520860 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:42.520872 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:42.520927 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.520998 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.521088 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":418,"threads_started":1}
I20260926 01:08:42.525569 19062 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:42.526154 19062 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:42.529956 19062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd: Generated new TSK 0
I20260926 01:08:42.530145 19062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:42.533334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34255
--builtin_ntp_servers=127.0.98.20:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:08:42.630174 19073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:42.630439 19073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:42.630474 19073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:42.631726 19073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:42.631771 19073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:42.631796 19073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:08:42.633221 19073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:42.633460 19073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:42.633767 19073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:42.639364 19078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.639380 19073 server_base.cc:1034] running on GCE node
W20260926 01:08:42.639472 19079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:42.639364 19081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.639890 19073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:42.640167 19073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:42.641330 19073 hybrid_clock.cc:648] HybridClock initialized: now 1790384922641315 us; error 34 us; skew 500 ppm
I20260926 01:08:42.642705 19073 webserver.cc:460] Webserver started at http://127.0.98.1:38829/ using document root <none> and password file <none>
I20260926 01:08:42.642947 19073 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:42.642998 19073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:42.643131 19073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:42.644229 19073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "7200aeabf7b54c999bc84443e08087fc"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "604b2ca33531bac9f010dbe9344d09d4"
server_key_iv: "570c554d24eb6976dddc453ac6aba4ae"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.644570 19073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7200aeabf7b54c999bc84443e08087fc"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "604b2ca33531bac9f010dbe9344d09d4"
server_key_iv: "570c554d24eb6976dddc453ac6aba4ae"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.645922 19073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:42.646916 19087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.647106 19073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:08:42.647205 19073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "7200aeabf7b54c999bc84443e08087fc"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "604b2ca33531bac9f010dbe9344d09d4"
server_key_iv: "570c554d24eb6976dddc453ac6aba4ae"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.647274 19073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:42.665210 19073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:42.665529 19073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:42.665658 19073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:42.665889 19073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:42.666210 19073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:42.666250 19073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.666281 19073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:42.666298 19073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.672227 19073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43507
I20260926 01:08:42.672293 19200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43507 every 8 connection(s)
I20260926 01:08:42.672638 19073 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:08:42.678056 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.673339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38123 (local address 127.0.98.62:34255)
0926 01:08:42.673399 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.673401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.673623 (+ 222us) server_negotiation.cc:407] Connection header received
0926 01:08:42.674156 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.674158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.674169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.674230 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:42.675004 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.675520 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.676541 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.676782 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.676850 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.677738 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:42.677739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:42.677739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:42.677745 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:42.677760 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:42.677764 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:42.677845 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.677951 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.677987 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:08:42.678196 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.673325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34255 (local address 127.0.98.1:38123)
0926 01:08:42.673505 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.673538 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.674038 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.674323 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.674326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.674336 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:42.674897 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.674902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.675616 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.675617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.676446 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.676448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.676875 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.676876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.677005 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.677582 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:08:42.677583 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:42.677615 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:08:42.677639 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:42.677851 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:42.677855 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:42.677858 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.677908 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.677999 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260926 01:08:42.678717 19201 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34255
I20260926 01:08:42.678812 19201 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:42.678992 19201 heartbeater.cc:511] Master 127.0.98.62:34255 requested a full tablet report, sending...
I20260926 01:08:42.679119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 19073
I20260926 01:08:42.679211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:08:42.679451 392 external_mini_cluster.cc:1428] Setting key 4a6106891f1b90e3da3af1c31e6723fe
I20260926 01:08:42.679569 19012 ts_manager.cc:194] Registered new tserver with Master: 7200aeabf7b54c999bc84443e08087fc (127.0.98.1:43507)
I20260926 01:08:42.680538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34255
--builtin_ntp_servers=127.0.98.20:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:08:42.680557 19012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38123
W20260926 01:08:42.764308 19204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:42.764657 19204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:42.764699 19204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:42.766065 19204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:42.766125 19204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:42.766155 19204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:08:42.767771 19204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:42.768026 19204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:42.768312 19204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:42.773816 19212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:42.773828 19209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:42.773828 19210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.774068 19204 server_base.cc:1034] running on GCE node
I20260926 01:08:42.774335 19204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:42.774616 19204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:42.775792 19204 hybrid_clock.cc:648] HybridClock initialized: now 1790384922775756 us; error 51 us; skew 500 ppm
I20260926 01:08:42.777324 19204 webserver.cc:460] Webserver started at http://127.0.98.2:38861/ using document root <none> and password file <none>
I20260926 01:08:42.777580 19204 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:42.777633 19204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:42.777769 19204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:42.778935 19204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "684b64d889ee4504a9d95c971deaea26"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "7eafc286f84f29ebe98aaac44c9dbf84"
server_key_iv: "13208360776191b23f490534b3542c45"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.779289 19204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "684b64d889ee4504a9d95c971deaea26"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "7eafc286f84f29ebe98aaac44c9dbf84"
server_key_iv: "13208360776191b23f490534b3542c45"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.780777 19204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 01:08:42.781646 19218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.781929 19204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:42.782029 19204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "684b64d889ee4504a9d95c971deaea26"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "7eafc286f84f29ebe98aaac44c9dbf84"
server_key_iv: "13208360776191b23f490534b3542c45"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.782100 19204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:42.804139 19204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:42.804472 19204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:42.804605 19204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:42.804824 19204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:42.805141 19204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:42.805192 19204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.805236 19204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:42.805254 19204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.810766 19204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41567
I20260926 01:08:42.810847 19331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41567 every 8 connection(s)
I20260926 01:08:42.811123 19204 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:08:42.815570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 19204
I20260926 01:08:42.815668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:08:42.815996 392 external_mini_cluster.cc:1428] Setting key 5485e8acd26503c1c3a080ee66b795ae
I20260926 01:08:42.816685 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.811913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40749 (local address 127.0.98.62:34255)
0926 01:08:42.811978 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.811980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.812181 (+ 201us) server_negotiation.cc:407] Connection header received
0926 01:08:42.812724 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.812725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.812743 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.812842 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:42.813541 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.814051 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.815072 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.815270 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.815335 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.816384 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:42.816386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:42.816387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:42.816393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:42.816412 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:42.816451 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:42.816492 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.816568 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.816600 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 01:08:42.816942 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.811834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34255 (local address 127.0.98.2:40749)
0926 01:08:42.812064 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.812100 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.812615 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.812924 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.812928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.812941 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:42.813450 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.813455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.814142 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.814143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.814987 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.814989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.815387 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.815388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.815539 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.816209 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:08:42.816210 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:42.816244 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:08:42.816272 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:42.816506 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:42.816511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:42.816515 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.816570 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.816674 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260926 01:08:42.817071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34255
--builtin_ntp_servers=127.0.98.20:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:08:42.817657 19332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34255
I20260926 01:08:42.817762 19332 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:42.817957 19332 heartbeater.cc:511] Master 127.0.98.62:34255 requested a full tablet report, sending...
I20260926 01:08:42.818531 19012 ts_manager.cc:194] Registered new tserver with Master: 684b64d889ee4504a9d95c971deaea26 (127.0.98.2:41567)
I20260926 01:08:42.818917 19012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40749
W20260926 01:08:42.900665 19335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:42.900910 19335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:42.900934 19335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:42.902127 19335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:42.902168 19335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:42.902194 19335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:08:42.903680 19335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:42.903925 19335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:42.904206 19335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:42.909849 19341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.909875 19335 server_base.cc:1034] running on GCE node
W20260926 01:08:42.909822 19340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:42.909822 19343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:42.910211 19335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:42.910538 19335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:42.911713 19335 hybrid_clock.cc:648] HybridClock initialized: now 1790384922911700 us; error 29 us; skew 500 ppm
I20260926 01:08:42.913128 19335 webserver.cc:460] Webserver started at http://127.0.98.3:35507/ using document root <none> and password file <none>
I20260926 01:08:42.913395 19335 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:42.913450 19335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:42.913628 19335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:42.915117 19335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "c25cefc7fbd44f13b9a2e467e51b2df5"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "400547ca51deeddfba285977e4cf90f0"
server_key_iv: "faf80d66747ef8cb7a440c9faecf6258"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.915759 19335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c25cefc7fbd44f13b9a2e467e51b2df5"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "400547ca51deeddfba285977e4cf90f0"
server_key_iv: "faf80d66747ef8cb7a440c9faecf6258"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.917234 19335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:08:42.918182 19349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.918478 19335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:42.918572 19335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "c25cefc7fbd44f13b9a2e467e51b2df5"
format_stamp: "Formatted at 2026-09-26 01:08:42 on dist-test-slave-l73v"
server_key: "400547ca51deeddfba285977e4cf90f0"
server_key_iv: "faf80d66747ef8cb7a440c9faecf6258"
server_key_version: "encryptionkey@0"
I20260926 01:08:42.918747 19335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:42.941917 19335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:42.942273 19335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:42.942457 19335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:42.942698 19335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:42.943034 19335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:42.943068 19335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.943099 19335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:42.943115 19335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:42.948937 19335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39509
I20260926 01:08:42.948977 19462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39509 every 8 connection(s)
I20260926 01:08:42.949316 19335 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:08:42.952996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 19335
I20260926 01:08:42.953099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:08:42.953301 392 external_mini_cluster.cc:1428] Setting key 6a2f6de07bf4c7f59002735dcee5bada
I20260926 01:08:42.954865 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.950047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39137 (local address 127.0.98.62:34255)
0926 01:08:42.950145 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.950147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.950473 (+ 326us) server_negotiation.cc:407] Connection header received
0926 01:08:42.951112 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.951113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.951126 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.951179 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:42.951849 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.952354 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.953360 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.953635 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.953693 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.954604 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:42.954605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:42.954606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:42.954611 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:42.954627 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:42.954632 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:42.954675 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.954749 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.954783 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 01:08:42.955092 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.950036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34255 (local address 127.0.98.3:39137)
0926 01:08:42.950328 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.950364 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.950980 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.951253 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.951257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.951267 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:42.951763 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.951768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.952446 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.952447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.953279 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.953281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.953713 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.953714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.953846 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.954452 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:08:42.954453 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:42.954484 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:08:42.954510 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:42.954680 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:42.954686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:42.954690 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.954754 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.954866 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
I20260926 01:08:42.955749 19463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34255
I20260926 01:08:42.955847 19463 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:42.956038 19463 heartbeater.cc:511] Master 127.0.98.62:34255 requested a full tablet report, sending...
I20260926 01:08:42.956485 19012 ts_manager.cc:194] Registered new tserver with Master: c25cefc7fbd44f13b9a2e467e51b2df5 (127.0.98.3:39509)
I20260926 01:08:42.956881 19012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39137
I20260926 01:08:42.965166 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:08:42.970319 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.966962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.0.98.62:34255)
0926 01:08:42.967060 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.967062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.967246 (+ 184us) server_negotiation.cc:407] Connection header received
0926 01:08:42.967282 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.967284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.967295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.967349 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:42.967555 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.968058 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.968852 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.969092 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.969153 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.969907 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:42.969908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:42.969908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:42.969913 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:42.969928 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:42.969933 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:42.970045 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.970211 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.970228 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 01:08:42.973188 19012 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:08:42.974912 19012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:08:42.981864 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.977787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.0.98.2:41567)
0926 01:08:42.977958 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.977962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.978449 (+ 487us) server_negotiation.cc:407] Connection header received
0926 01:08:42.978494 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.978496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.978512 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.978577 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:42.978757 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.979931 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.980992 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.981516 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.981585 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.981669 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.981736 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.981777 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260926 01:08:42.981868 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.977443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39509 (local address 127.0.0.1:40790)
0926 01:08:42.977683 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.977691 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.977728 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.978029 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.978032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.978034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:42.978163 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.978165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.979627 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.979628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.980763 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.980765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.981483 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.981485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.981567 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.981583 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.981648 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.981754 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":3,"thread_start_us":60,"threads_started":1}
I20260926 01:08:42.982111 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.977669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.0.98.3:39509)
0926 01:08:42.977859 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.977865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.977876 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:08:42.977918 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.977920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.977936 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.978025 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:42.978245 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.979543 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.980862 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.981403 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.981459 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.981529 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.981991 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.982038 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260926 01:08:42.981875 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.977557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41567 (local address 127.0.0.1:45806)
0926 01:08:42.978186 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.978197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.978248 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.978642 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.978643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.978645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:42.978694 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.978696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.979998 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.979999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.980914 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.980917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.981599 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.981600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.981660 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.981665 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.981723 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.981763 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":80,"threads_started":1}
I20260926 01:08:42.982686 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.977360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43507 (local address 127.0.0.1:36472)
0926 01:08:42.977566 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:42.977627 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:42.977707 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:42.978302 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:42.978303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:42.978304 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:42.978348 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.978350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.979449 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.979450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.980763 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:42.980765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.982428 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.982429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:42.982502 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.982507 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:08:42.982568 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 01:08:42.982613 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260926 01:08:42.982815 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:42.977478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.0.98.1:43507)
0926 01:08:42.977973 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:42.977976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:42.977986 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:08:42.978020 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:42.978022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:42.978035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:42.978105 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:42.978436 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.979371 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.981565 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:42.982327 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:42.982418 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:42.982514 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:42.982686 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 01:08:42.982731 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":102,"threads_started":1}
I20260926 01:08:42.983002 19397 tablet_service.cc:1467] Processing CreateTablet for tablet 82f4ac8dad8b461690b0873ad10b52ba (DEFAULT_TABLE table=test-table [id=dc536f8309704f86ad2e9fe79bc48672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:42.983393 19397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f4ac8dad8b461690b0873ad10b52ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:42.983389 19266 tablet_service.cc:1467] Processing CreateTablet for tablet 82f4ac8dad8b461690b0873ad10b52ba (DEFAULT_TABLE table=test-table [id=dc536f8309704f86ad2e9fe79bc48672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:42.983687 19266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f4ac8dad8b461690b0873ad10b52ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:42.987030 19478 tablet_bootstrap.cc:492] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: Bootstrap starting.
I20260926 01:08:42.986974 19135 tablet_service.cc:1467] Processing CreateTablet for tablet 82f4ac8dad8b461690b0873ad10b52ba (DEFAULT_TABLE table=test-table [id=dc536f8309704f86ad2e9fe79bc48672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:42.987398 19135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f4ac8dad8b461690b0873ad10b52ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:42.987742 19478 tablet_bootstrap.cc:654] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:42.988162 19478 log.cc:824] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:42.988700 19477 tablet_bootstrap.cc:492] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26: Bootstrap starting.
I20260926 01:08:42.989065 19478 tablet_bootstrap.cc:492] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: No bootstrap required, opened a new log
I20260926 01:08:42.989148 19478 ts_tablet_manager.cc:1397] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:08:42.989553 19477 tablet_bootstrap.cc:654] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:42.989900 19477 log.cc:824] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:42.990765 19477 tablet_bootstrap.cc:492] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26: No bootstrap required, opened a new log
I20260926 01:08:42.990849 19477 ts_tablet_manager.cc:1397] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:08:42.991321 19478 raft_consensus.cc:348] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 } } peers { permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 } } peers { permanent_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } }
I20260926 01:08:42.991509 19478 raft_consensus.cc:374] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:42.991559 19478 raft_consensus.cc:729] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25cefc7fbd44f13b9a2e467e51b2df5, State: Initialized, Role: FOLLOWER
I20260926 01:08:42.991456 19481 tablet_bootstrap.cc:492] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc: Bootstrap starting.
I20260926 01:08:42.991678 19478 consensus_queue.cc:260] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 } } peers { permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 } } peers { permanent_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } }
I20260926 01:08:42.991967 19478 ts_tablet_manager.cc:1428] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:42.992100 19463 heartbeater.cc:503] Master 127.0.98.62:34255 was elected leader, sending a full tablet report...
I20260926 01:08:42.992552 19481 tablet_bootstrap.cc:654] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:42.992940 19481 log.cc:824] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:42.993805 19481 tablet_bootstrap.cc:492] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc: No bootstrap required, opened a new log
I20260926 01:08:42.993886 19481 ts_tablet_manager.cc:1397] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:08:42.993988 19477 raft_consensus.cc:348] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 } } peers { permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 } } peers { permanent_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } }
I20260926 01:08:42.994143 19477 raft_consensus.cc:374] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:42.994174 19477 raft_consensus.cc:729] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684b64d889ee4504a9d95c971deaea26, State: Initialized, Role: FOLLOWER
I20260926 01:08:42.994297 19477 consensus_queue.cc:260] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 } } peers { permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 } } peers { permanent_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } }
I20260926 01:08:42.994575 19477 ts_tablet_manager.cc:1428] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 01:08:42.994637 19332 heartbeater.cc:503] Master 127.0.98.62:34255 was elected leader, sending a full tablet report...
I20260926 01:08:42.996517 19481 raft_consensus.cc:348] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 } } peers { permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 } } peers { permanent_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } }
I20260926 01:08:42.996675 19481 raft_consensus.cc:374] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:42.996716 19481 raft_consensus.cc:729] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7200aeabf7b54c999bc84443e08087fc, State: Initialized, Role: FOLLOWER
I20260926 01:08:42.996845 19481 consensus_queue.cc:260] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 } } peers { permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 } } peers { permanent_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } }
I20260926 01:08:42.997097 19481 ts_tablet_manager.cc:1428] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:42.997141 19201 heartbeater.cc:503] Master 127.0.98.62:34255 was elected leader, sending a full tablet report...
I20260926 01:08:43.027325 19482 raft_consensus.cc:482] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:08:43.027465 19482 raft_consensus.cc:504] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 } } peers { permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 } } peers { permanent_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } }
I20260926 01:08:43.027813 19482 leader_election.cc:288] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7200aeabf7b54c999bc84443e08087fc (127.0.98.1:43507), 684b64d889ee4504a9d95c971deaea26 (127.0.98.2:41567)
I20260926 01:08:43.032321 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:43.028006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43507 (local address 127.0.98.3:42933)
0926 01:08:43.028091 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:43.028104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:43.028155 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:43.028616 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:43.028617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:43.028620 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:43.028688 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:43.028690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:43.029634 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:43.029635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:43.030970 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:43.030974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:43.031983 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:43.031990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:43.032079 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:43.032086 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 01:08:43.032156 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 01:08:43.032211 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":68}
I20260926 01:08:43.032346 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:43.028370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48927 (local address 127.0.98.2:41567)
0926 01:08:43.028427 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:43.028429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:43.028441 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:08:43.028472 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:43.028473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:43.028485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:43.028556 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:43.028868 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:43.029560 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:43.031094 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:43.031813 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:43.031902 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:43.031913 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:43.032217 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 01:08:43.032273 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 01:08:43.032346 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:43.028392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42933 (local address 127.0.98.1:43507)
0926 01:08:43.028444 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:43.028446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:43.028453 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:08:43.028475 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:43.028476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:43.028485 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:43.028556 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:43.028763 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:43.029524 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:43.031095 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:43.031815 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:43.031902 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:43.031913 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:43.032217 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 01:08:43.032268 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:08:43.032330 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:43.028138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41567 (local address 127.0.98.3:48927)
0926 01:08:43.028316 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:43.028324 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:43.028360 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:43.028765 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:43.028765 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:43.028767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:43.028807 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:43.028807 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:43.029645 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:43.029646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:43.030970 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:43.030974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:43.031983 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:43.031984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:43.032079 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:43.032086 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 01:08:43.032158 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 01:08:43.032209 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":82,"thread_start_us":93,"threads_started":1}
I20260926 01:08:43.032902 19286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f4ac8dad8b461690b0873ad10b52ba" candidate_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684b64d889ee4504a9d95c971deaea26" is_pre_election: true
I20260926 01:08:43.033190 19286 raft_consensus.cc:2454] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25cefc7fbd44f13b9a2e467e51b2df5 in term 0.
I20260926 01:08:43.033164 19155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f4ac8dad8b461690b0873ad10b52ba" candidate_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7200aeabf7b54c999bc84443e08087fc" is_pre_election: true
I20260926 01:08:43.033303 19155 raft_consensus.cc:2454] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25cefc7fbd44f13b9a2e467e51b2df5 in term 0.
I20260926 01:08:43.033465 19352 leader_election.cc:302] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684b64d889ee4504a9d95c971deaea26, c25cefc7fbd44f13b9a2e467e51b2df5; no voters:
I20260926 01:08:43.033610 19482 raft_consensus.cc:2781] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:08:43.033676 19482 raft_consensus.cc:482] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:08:43.033697 19482 raft_consensus.cc:3037] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:43.034454 19482 raft_consensus.cc:504] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 } } peers { permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 } } peers { permanent_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } }
I20260926 01:08:43.034583 19482 leader_election.cc:288] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [CANDIDATE]: Term 1 election: Requested vote from peers 7200aeabf7b54c999bc84443e08087fc (127.0.98.1:43507), 684b64d889ee4504a9d95c971deaea26 (127.0.98.2:41567)
I20260926 01:08:43.034832 19155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f4ac8dad8b461690b0873ad10b52ba" candidate_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7200aeabf7b54c999bc84443e08087fc"
I20260926 01:08:43.034852 19286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f4ac8dad8b461690b0873ad10b52ba" candidate_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684b64d889ee4504a9d95c971deaea26"
I20260926 01:08:43.034942 19286 raft_consensus.cc:3037] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:43.034942 19155 raft_consensus.cc:3037] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:43.035885 19286 raft_consensus.cc:2454] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25cefc7fbd44f13b9a2e467e51b2df5 in term 1.
I20260926 01:08:43.035904 19155 raft_consensus.cc:2454] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25cefc7fbd44f13b9a2e467e51b2df5 in term 1.
I20260926 01:08:43.036106 19352 leader_election.cc:302] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7200aeabf7b54c999bc84443e08087fc, c25cefc7fbd44f13b9a2e467e51b2df5; no voters:
I20260926 01:08:43.036211 19482 raft_consensus.cc:2781] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:43.036363 19482 raft_consensus.cc:686] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 1 LEADER]: Becoming Leader. State: Replica: c25cefc7fbd44f13b9a2e467e51b2df5, State: Running, Role: LEADER
I20260926 01:08:43.036446 19482 consensus_queue.cc:237] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 } } peers { permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 } } peers { permanent_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } }
I20260926 01:08:43.037374 19013 catalog_manager.cc:5495] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c25cefc7fbd44f13b9a2e467e51b2df5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c25cefc7fbd44f13b9a2e467e51b2df5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39509 } health_report { overall_health: HEALTHY } } }
I20260926 01:08:43.043833 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:43.041412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.0.98.3:39509)
0926 01:08:43.041505 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:43.041507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:43.041516 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:08:43.041583 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:43.041583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:43.041604 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:43.041654 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:08:43.041750 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:43.042207 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:43.043010 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:43.043211 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:43.043276 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:43.043444 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:08:43.043580 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:08:43.043678 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:43.043709 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 01:08:43.043717 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 01:08:43.046078 19286 raft_consensus.cc:1261] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26 [term 1 FOLLOWER]: Refusing update from remote peer c25cefc7fbd44f13b9a2e467e51b2df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:43.046505 19487 consensus_queue.cc:1035] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "684b64d889ee4504a9d95c971deaea26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:43.046911 19155 raft_consensus.cc:1261] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc [term 1 FOLLOWER]: Refusing update from remote peer c25cefc7fbd44f13b9a2e467e51b2df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:43.047302 19487 consensus_queue.cc:1035] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7200aeabf7b54c999bc84443e08087fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:43.048949 19490 mvcc.cc:205] Tried to move back new op lower bound from 7333416644794089472 to 7333416644758360064. Current Snapshot: MvccSnapshot[applied={T|T < 7333416644794089472}]
I20260926 01:08:43.049505 19491 mvcc.cc:205] Tried to move back new op lower bound from 7333416644794089472 to 7333416644758360064. Current Snapshot: MvccSnapshot[applied={T|T < 7333416644794089472}]
I20260926 01:08:43.050105 19493 mvcc.cc:205] Tried to move back new op lower bound from 7333416644794089472 to 7333416644758360064. Current Snapshot: MvccSnapshot[applied={T|T < 7333416644794089472}]
I20260926 01:08:43.051386 19013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:43.051496 19013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:43.052237 19013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e42615d3a5494eb1a8ebb569f3b66ebd: Sending DeleteTablet for 3 replicas of tablet 82f4ac8dad8b461690b0873ad10b52ba
I20260926 01:08:43.052570 19135 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f4ac8dad8b461690b0873ad10b52ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:43 UTC) from {username='slave'} at 127.0.0.1:36472
I20260926 01:08:43.052574 19397 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f4ac8dad8b461690b0873ad10b52ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:43 UTC) from {username='slave'} at 127.0.0.1:40790
I20260926 01:08:43.052639 19266 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f4ac8dad8b461690b0873ad10b52ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:43 UTC) from {username='slave'} at 127.0.0.1:45806
I20260926 01:08:43.052901 19501 tablet_replica.cc:330] T 82f4ac8dad8b461690b0873ad10b52ba P 7200aeabf7b54c999bc84443e08087fc: stopping tablet replica
I20260926 01:08:43.053218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 19073
I20260926 01:08:43.055680 19500 tablet_replica.cc:330] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: stopping tablet replica
I20260926 01:08:43.055802 19500 raft_consensus.cc:2229] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:08:43.055919 19500 raft_consensus.cc:2258] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:43.058627 19502 tablet_replica.cc:330] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26: stopping tablet replica
I20260926 01:08:43.058789 19502 raft_consensus.cc:2229] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:08:43.058918 19502 raft_consensus.cc:2258] T 82f4ac8dad8b461690b0873ad10b52ba P 684b64d889ee4504a9d95c971deaea26 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:08:43.059074 18999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43507 (error 108)
I20260926 01:08:43.059304 19500 ts_tablet_manager.cc:1905] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:43.059561 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:43.059445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43507 (local address 127.0.0.1:36474)
0926 01:08:43.059485 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:43.059525 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 01:08:43.059661 18999 catalog_manager.cc:4584] TS 7200aeabf7b54c999bc84443e08087fc (127.0.98.1:43507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82f4ac8dad8b461690b0873ad10b52ba: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43507: connect: Connection refused (error 111)
I20260926 01:08:43.059926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 19204
I20260926 01:08:43.063802 19500 ts_tablet_manager.cc:1918] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:43.063905 19500 log.cc:1196] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/82f4ac8dad8b461690b0873ad10b52ba
I20260926 01:08:43.064167 19500 ts_tablet_manager.cc:1939] T 82f4ac8dad8b461690b0873ad10b52ba P c25cefc7fbd44f13b9a2e467e51b2df5: Deleting consensus metadata
I20260926 01:08:43.064723 18997 catalog_manager.cc:4841] TS c25cefc7fbd44f13b9a2e467e51b2df5 (127.0.98.3:39509): tablet 82f4ac8dad8b461690b0873ad10b52ba (table test-table [id=dc536f8309704f86ad2e9fe79bc48672]) successfully deleted
I20260926 01:08:43.068346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 19335
I20260926 01:08:43.068560 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:43.068402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41567 (local address 127.0.0.1:45818)
0926 01:08:43.068441 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:43.068475 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 01:08:43.073474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 18982
2026-09-26T01:08:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (719 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:43 dist-test-slave-l73v krb5kdc[19506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:43 dist-test-slave-l73v krb5kdc[19506](info): set up 2 sockets
Sep 26 01:08:43 dist-test-slave-l73v krb5kdc[19506](info): commencing operation
krb5kdc: starting...
W20260926 01:08:44.042029 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:44 dist-test-slave-l73v krb5kdc[19506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:08:44.165848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:44.248071 19519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:44.248314 19519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:44.248339 19519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:44.249625 19519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:44.249682 19519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:44.249696 19519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:44.249706 19519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:44.249718 19519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:44.251354 19519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:44.251621 19519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:44.251853 19519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:44.256804 19527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:44.256808 19525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:44.257095 19519 server_base.cc:1034] running on GCE node
W20260926 01:08:44.256804 19524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:44.257395 19519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:44.257704 19519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:44.258883 19519 hybrid_clock.cc:648] HybridClock initialized: now 1790384924258869 us; error 31 us; skew 500 ppm
Sep 26 01:08:44 dist-test-slave-l73v krb5kdc[19506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:44.261631 19519 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:08:44.262137 19519 webserver.cc:460] Webserver started at http://127.0.98.62:37217/ using document root <none> and password file <none>
I20260926 01:08:44.262338 19519 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:44.262408 19519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:44.262539 19519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:44.263614 19519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "8c159ec23f824a15a305ffe908cd1574"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "9083437861046177c0463b4b4087424c"
server_key_iv: "e2174599111309ba0ad44cf05d8716c7"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.263965 19519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c159ec23f824a15a305ffe908cd1574"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "9083437861046177c0463b4b4087424c"
server_key_iv: "e2174599111309ba0ad44cf05d8716c7"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.265705 19519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:44.266789 19534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.266983 19519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:44.267081 19519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "8c159ec23f824a15a305ffe908cd1574"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "9083437861046177c0463b4b4087424c"
server_key_iv: "e2174599111309ba0ad44cf05d8716c7"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.267151 19519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:44.291242 19519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:44.293488 19519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:44.293634 19519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:44.297940 19519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44379
I20260926 01:08:44.298018 19586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44379 every 8 connection(s)
I20260926 01:08:44.298348 19519 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:44.299333 19587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:44.301133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 19519
I20260926 01:08:44.301246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:44.301429 392 external_mini_cluster.cc:1428] Setting key baa969524b2e4b5dea6c11616aad6866
I20260926 01:08:44.301852 19587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574: Bootstrap starting.
I20260926 01:08:44.302520 19587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:44.302807 19587 log.cc:824] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:44.303617 19587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574: No bootstrap required, opened a new log
I20260926 01:08:44.305706 19587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c159ec23f824a15a305ffe908cd1574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260926 01:08:44.305867 19587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:44.305891 19587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c159ec23f824a15a305ffe908cd1574, State: Initialized, Role: FOLLOWER
I20260926 01:08:44.306007 19587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c159ec23f824a15a305ffe908cd1574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260926 01:08:44.306077 19587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:44.306128 19587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:44.306175 19587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:44.306967 19587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c159ec23f824a15a305ffe908cd1574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260926 01:08:44.307088 19587 leader_election.cc:302] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c159ec23f824a15a305ffe908cd1574; no voters:
I20260926 01:08:44.307282 19587 leader_election.cc:288] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:44.307330 19592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:44.307538 19592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [term 1 LEADER]: Becoming Leader. State: Replica: 8c159ec23f824a15a305ffe908cd1574, State: Running, Role: LEADER
I20260926 01:08:44.307575 19587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:44.307642 19592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c159ec23f824a15a305ffe908cd1574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260926 01:08:44.308116 19593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c159ec23f824a15a305ffe908cd1574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c159ec23f824a15a305ffe908cd1574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } } }
I20260926 01:08:44.308218 19593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [sys.catalog]: This master's current role is: LEADER
Sep 26 01:08:44 dist-test-slave-l73v krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:44.309016 19594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c159ec23f824a15a305ffe908cd1574. Latest consensus state: current_term: 1 leader_uuid: "8c159ec23f824a15a305ffe908cd1574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c159ec23f824a15a305ffe908cd1574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } } }
I20260926 01:08:44.309180 19594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574 [sys.catalog]: This master's current role is: LEADER
W20260926 01:08:44.309532 19608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:08:44.309584 19608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:08:44.309638 19599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:44.309825 19599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:44.311508 19599 catalog_manager.cc:1347] Generated new cluster ID: cdd8bbeed28f45259750d6df9a2977f6
I20260926 01:08:44.311554 19599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:44.312988 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.301979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.98.62:44379)
0926 01:08:44.302165 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.302173 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.302200 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:08:44.302730 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.302738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.302790 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.302944 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:44.303640 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.304621 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.305726 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.306278 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.306372 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.309354 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:44.309358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:44.309359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:44.309371 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:44.311336 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:44.311737 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:44.311740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:44.311741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:44.311845 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:44.312044 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:44.312045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:44.312045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:44.312358 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:44.312512 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.312609 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.312755 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260926 01:08:44.315058 19599 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:44.315739 19599 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:44.320379 19599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574: Generated new TSK 0
I20260926 01:08:44.320603 19599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 01:08:44.350203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:44.434094 19613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:44.434322 19613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:44.434357 19613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:44.435628 19613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:44.435680 19613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:44.435719 19613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:08:44.437299 19613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:44.437564 19613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:44.437790 19613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:44.442943 19618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:44.442858 19619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:44.442873 19621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:44.443207 19613 server_base.cc:1034] running on GCE node
I20260926 01:08:44.443426 19613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:44.443681 19613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:44.444859 19613 hybrid_clock.cc:648] HybridClock initialized: now 1790384924444838 us; error 29 us; skew 500 ppm
Sep 26 01:08:44 dist-test-slave-l73v krb5kdc[19506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:44.447541 19613 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 01:08:44.447988 19613 webserver.cc:460] Webserver started at http://127.0.98.1:44581/ using document root <none> and password file <none>
I20260926 01:08:44.448197 19613 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:44.448246 19613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:44.448372 19613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:44.449432 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "a033f5b4a6dd43e1a73888f44a9d3a89"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "2cd1e3c162c8e5e40753f97e41acfa83"
server_key_iv: "caaa43e8e65e56449d6c1e29ef5dca78"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.449764 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a033f5b4a6dd43e1a73888f44a9d3a89"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "2cd1e3c162c8e5e40753f97e41acfa83"
server_key_iv: "caaa43e8e65e56449d6c1e29ef5dca78"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.451149 19613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:08:44.451897 19628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.452070 19613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:08:44.452153 19613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "a033f5b4a6dd43e1a73888f44a9d3a89"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "2cd1e3c162c8e5e40753f97e41acfa83"
server_key_iv: "caaa43e8e65e56449d6c1e29ef5dca78"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.452217 19613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:44.474841 19613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:44.477118 19613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:44.477267 19613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:44.477499 19613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:44.477837 19613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:44.477895 19613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.477927 19613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:44.477950 19613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.483640 19613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46327
I20260926 01:08:44.483721 19741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46327 every 8 connection(s)
I20260926 01:08:44.484012 19613 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:08:44.485652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 19613
I20260926 01:08:44.485759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:08:44.485978 392 external_mini_cluster.cc:1428] Setting key 06fbc9eb48e2cfce2d79d3546b86d0a9
Sep 26 01:08:44 dist-test-slave-l73v krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 01:08:44.494341 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.484782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49227 (local address 127.0.98.62:44379)
0926 01:08:44.484868 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.484870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.485101 (+ 231us) server_negotiation.cc:407] Connection header received
0926 01:08:44.486087 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.486089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.486102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.486173 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:44.486997 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.487730 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.488839 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.489107 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.489170 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.491569 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:44.491571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:44.491571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:44.491579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:44.492980 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:44.493377 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:44.493378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:44.493379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:44.493434 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:44.493658 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:44.493659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:44.493659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:44.493792 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:44.493858 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.494186 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.494271 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 01:08:44.494585 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.484742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44379 (local address 127.0.98.1:49227)
0926 01:08:44.484953 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:44.484997 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:44.485972 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:44.486274 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:44.486278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:44.486662 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:44.486903 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.486908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.487840 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.487841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.488743 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.488745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.489210 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.489211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.489369 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.489946 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:44.489955 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:44.491468 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:44.493054 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:44.493064 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:44.493067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:44.493307 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:44.493489 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:44.493489 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:44.493490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:44.493602 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:44.493867 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:44.493875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:44.494063 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 01:08:44.494169 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 01:08:44.494319 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260926 01:08:44.495298 19742 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44379
I20260926 01:08:44.495399 19742 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:44.495580 19742 heartbeater.cc:511] Master 127.0.98.62:44379 requested a full tablet report, sending...
I20260926 01:08:44.496085 19551 ts_manager.cc:194] Registered new tserver with Master: a033f5b4a6dd43e1a73888f44a9d3a89 (127.0.98.1:46327)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 01:08:44.496781 19551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 01:08:44.515139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:44.598665 19747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:44.598968 19747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:44.599004 19747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:44.600474 19747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:44.600528 19747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:44.600567 19747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:08:44.602236 19747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:44.602548 19747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:44.602789 19747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:44.608011 19752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:44.608134 19747 server_base.cc:1034] running on GCE node
W20260926 01:08:44.608011 19753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:44.608021 19755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:44.608552 19747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:44.608846 19747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:44.610006 19747 hybrid_clock.cc:648] HybridClock initialized: now 1790384924609992 us; error 29 us; skew 500 ppm
Sep 26 01:08:44 dist-test-slave-l73v krb5kdc[19506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:44.612551 19747 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 01:08:44.613020 19747 webserver.cc:460] Webserver started at http://127.0.98.2:32803/ using document root <none> and password file <none>
I20260926 01:08:44.613293 19747 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:44.613346 19747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:44.613474 19747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:44.614600 19747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "607f3fbf7b2041c1b0b4a216232b2ba8"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "0827352572af3b92924a23f43c727ff5"
server_key_iv: "9d76ee4aa4dc59a2227cbdb94e2a502a"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.614928 19747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "607f3fbf7b2041c1b0b4a216232b2ba8"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "0827352572af3b92924a23f43c727ff5"
server_key_iv: "9d76ee4aa4dc59a2227cbdb94e2a502a"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.616178 19747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:44.616890 19762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.617043 19747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:08:44.617125 19747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "607f3fbf7b2041c1b0b4a216232b2ba8"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "0827352572af3b92924a23f43c727ff5"
server_key_iv: "9d76ee4aa4dc59a2227cbdb94e2a502a"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.617185 19747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:44.634222 19747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:44.636530 19747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:44.636693 19747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:44.636929 19747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:44.637259 19747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:44.637300 19747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.637331 19747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:44.637348 19747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.643211 19747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46659
I20260926 01:08:44.643262 19875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46659 every 8 connection(s)
I20260926 01:08:44.643594 19747 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 01:08:44 dist-test-slave-l73v krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:44.650535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 19747
I20260926 01:08:44.650662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:08:44.650879 392 external_mini_cluster.cc:1428] Setting key 220d1f0f588511b8b86009de165855df
I20260926 01:08:44.653699 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.644355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42721 (local address 127.0.98.62:44379)
0926 01:08:44.644408 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.644410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.644604 (+ 194us) server_negotiation.cc:407] Connection header received
0926 01:08:44.645559 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.645560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.645571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.645645 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:44.646417 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.646952 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.648042 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.648235 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.648297 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.650931 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:44.650933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:44.650934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:44.650943 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:44.652243 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:44.652746 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:44.652747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:44.652748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:44.652811 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:44.653031 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:44.653032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:44.653033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:44.653170 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:44.653233 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.653513 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.653608 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:08:44.653843 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.644308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44379 (local address 127.0.98.2:42721)
0926 01:08:44.644484 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:44.644527 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:44.645434 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:44.645780 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:44.645784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:44.646069 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:44.646281 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.646286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.647053 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.647054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.647948 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.647950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.648351 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.648352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.648475 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.649016 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:44.649025 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:44.650781 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:44.652329 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:44.652338 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:44.652340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:44.652664 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:44.652877 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:44.652877 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:44.652878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:44.652982 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:44.653230 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:44.653233 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:44.653385 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 01:08:44.653462 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 01:08:44.653597 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260926 01:08:44.654301 19876 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44379
I20260926 01:08:44.654443 19876 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:44.654649 19876 heartbeater.cc:511] Master 127.0.98.62:44379 requested a full tablet report, sending...
I20260926 01:08:44.655076 19551 ts_manager.cc:194] Registered new tserver with Master: 607f3fbf7b2041c1b0b4a216232b2ba8 (127.0.98.2:46659)
I20260926 01:08:44.655462 19551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42721
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 01:08:44.679181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:44.763933 19881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:44.764158 19881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:44.764183 19881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:44.765496 19881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:44.765553 19881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:44.765583 19881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:08:44.767391 19881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:44.767673 19881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:44.767907 19881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:44.774210 19887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:44.774264 19889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:44.774312 19886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:44.774647 19881 server_base.cc:1034] running on GCE node
I20260926 01:08:44.774839 19881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:44.775056 19881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:44.776217 19881 hybrid_clock.cc:648] HybridClock initialized: now 1790384924776168 us; error 64 us; skew 500 ppm
Sep 26 01:08:44 dist-test-slave-l73v krb5kdc[19506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:44.778786 19881 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 01:08:44.779258 19881 webserver.cc:460] Webserver started at http://127.0.98.3:36939/ using document root <none> and password file <none>
I20260926 01:08:44.779454 19881 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:44.779492 19881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:44.779584 19881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:44.780616 19881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "e2b698dfdf26fe4bd6378eea7d73beb9"
server_key_iv: "2758e236073c437b8957dfa72ae5c143"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.780913 19881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "e2b698dfdf26fe4bd6378eea7d73beb9"
server_key_iv: "2758e236073c437b8957dfa72ae5c143"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.782132 19881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:08:44.782863 19896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.783002 19881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.783083 19881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412"
format_stamp: "Formatted at 2026-09-26 01:08:44 on dist-test-slave-l73v"
server_key: "e2b698dfdf26fe4bd6378eea7d73beb9"
server_key_iv: "2758e236073c437b8957dfa72ae5c143"
server_key_version: "encryptionkey@0"
I20260926 01:08:44.783139 19881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:44.809777 19881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:44.811940 19881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:44.812059 19881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:44.812252 19881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:44.812520 19881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:44.812549 19881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.812569 19881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:44.812587 19881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:44.818496 19881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34751
I20260926 01:08:44.818629 20009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34751 every 8 connection(s)
I20260926 01:08:44.818884 19881 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:08:44.824586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 19881
I20260926 01:08:44.824687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:08:44.824843 392 external_mini_cluster.cc:1428] Setting key c89cb2f5f50cd461fc1da4c057599493
Sep 26 01:08:44 dist-test-slave-l73v krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:08:44.828891 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.819597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34875 (local address 127.0.98.62:44379)
0926 01:08:44.819657 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.819659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.819867 (+ 208us) server_negotiation.cc:407] Connection header received
0926 01:08:44.820967 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.820969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.820980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.821068 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:44.821766 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.822305 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.823446 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.823662 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.823700 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.826192 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:44.826194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:44.826194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:44.826202 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:44.827567 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:44.828006 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:44.828007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:44.828008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:44.828067 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:44.828361 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:44.828362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:44.828362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:44.828473 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:44.828519 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.828759 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.828825 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:08:44.829102 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.819569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44379 (local address 127.0.98.3:34875)
0926 01:08:44.819742 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:44.819781 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:44.820859 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:44.821184 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:44.821188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:44.821465 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:44.821662 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.821666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.822451 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.822452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.823336 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.823338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.823778 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.823779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.823887 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.824398 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:08:44.824408 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:44.826078 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:44.827657 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:44.827667 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:44.827670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:44.827917 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:44.828156 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:08:44.828158 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:08:44.828158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:08:44.828303 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:08:44.828519 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:44.828522 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:44.828649 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 01:08:44.828744 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 01:08:44.828886 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260926 01:08:44.829561 20010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44379
I20260926 01:08:44.829694 20010 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:44.829850 20010 heartbeater.cc:511] Master 127.0.98.62:44379 requested a full tablet report, sending...
I20260926 01:08:44.830207 19551 ts_manager.cc:194] Registered new tserver with Master: ab7f1ed7a4b444f9b2b5e5cd310d8412 (127.0.98.3:34751)
I20260926 01:08:44.830621 19551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34875
I20260926 01:08:44.836921 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:08:44.844444 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.838207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.0.98.62:44379)
0926 01:08:44.838263 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.838264 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.838346 (+ 82us) server_negotiation.cc:407] Connection header received
0926 01:08:44.838443 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.838444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.838454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.838496 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:44.839053 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.839545 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.840332 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.840527 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.840644 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.841822 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:44.841824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:44.841824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:44.841831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:44.843209 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:44.843571 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:44.843572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:44.843572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:44.843668 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:44.843936 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:44.843936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:44.843937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:44.844109 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:44.844205 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.844306 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.844375 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:08:44.846292 19551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:08:44.848022 19551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:08:44.855257 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.850284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34751 (local address 127.0.0.1:54622)
0926 01:08:44.850637 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:44.850647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:44.850699 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:44.850988 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:44.850989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:44.850991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:44.851034 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.851036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.852426 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.852426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.854097 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.854099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.854925 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.854926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.855014 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.855031 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 01:08:44.855100 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 01:08:44.855162 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":35,"thread_start_us":62,"threads_started":1}
I20260926 01:08:44.855410 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.850418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.0.98.2:46659)
0926 01:08:44.850585 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.850589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.850598 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:08:44.850696 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.850699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.850714 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.850794 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:44.851008 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.852147 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.854190 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.854985 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.855057 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.855117 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.855289 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.855329 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":69,"threads_started":1}
I20260926 01:08:44.855621 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.850553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.0.98.3:34751)
0926 01:08:44.850853 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.850857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.850865 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:08:44.850900 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.850903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.850917 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.850991 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:44.851101 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.852161 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.854191 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.854845 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.854915 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.855022 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.855412 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.855495 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":81,"threads_started":1}
I20260926 01:08:44.855837 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.850284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46659 (local address 127.0.0.1:54278)
0926 01:08:44.850478 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:44.850519 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:44.850586 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:44.850816 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:44.850819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:44.850822 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:44.850903 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.850905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.852425 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.852426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.854074 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.854077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.855159 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.855160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.855205 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.855208 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:44.855257 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 01:08:44.855304 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":89,"thread_start_us":121,"threads_started":1}
I20260926 01:08:44.856326 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.850288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46327 (local address 127.0.0.1:56456)
0926 01:08:44.852402 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:44.852412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:44.852461 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:44.852668 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:44.852669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:44.852671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:44.852722 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.852723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.853945 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.853946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.855041 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.855044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.856054 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.856055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.856161 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.856165 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:44.856218 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 01:08:44.856262 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2085,"thread_start_us":130,"threads_started":1}
I20260926 01:08:44.856339 19944 tablet_service.cc:1467] Processing CreateTablet for tablet c7570178d66649db88e718cfc7087cab (DEFAULT_TABLE table=test-table [id=cc77781d3c2e4d06b2da3a16f994f484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:44.856552 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.850796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.0.98.1:46327)
0926 01:08:44.850952 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.850955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.852476 (+ 1521us) server_negotiation.cc:407] Connection header received
0926 01:08:44.852529 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.852532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.852548 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.852617 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:44.852789 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.853846 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.855113 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.855897 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.855964 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.856020 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.856246 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.856295 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260926 01:08:44.856712 19944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7570178d66649db88e718cfc7087cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:44.856628 19810 tablet_service.cc:1467] Processing CreateTablet for tablet c7570178d66649db88e718cfc7087cab (DEFAULT_TABLE table=test-table [id=cc77781d3c2e4d06b2da3a16f994f484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:44.856963 19810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7570178d66649db88e718cfc7087cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:44.857240 19676 tablet_service.cc:1467] Processing CreateTablet for tablet c7570178d66649db88e718cfc7087cab (DEFAULT_TABLE table=test-table [id=cc77781d3c2e4d06b2da3a16f994f484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:44.857560 19676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7570178d66649db88e718cfc7087cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:44.859606 20027 tablet_bootstrap.cc:492] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: Bootstrap starting.
I20260926 01:08:44.860244 20028 tablet_bootstrap.cc:492] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: Bootstrap starting.
I20260926 01:08:44.860322 20027 tablet_bootstrap.cc:654] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:44.860754 20027 log.cc:824] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:44.860911 20028 tablet_bootstrap.cc:654] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:44.860916 20029 tablet_bootstrap.cc:492] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89: Bootstrap starting.
I20260926 01:08:44.861421 20029 tablet_bootstrap.cc:654] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:44.861421 20028 log.cc:824] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:44.861840 20029 log.cc:824] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:44.862062 20028 tablet_bootstrap.cc:492] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: No bootstrap required, opened a new log
I20260926 01:08:44.862123 20028 ts_tablet_manager.cc:1397] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:08:44.862542 20029 tablet_bootstrap.cc:492] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89: No bootstrap required, opened a new log
I20260926 01:08:44.862574 20027 tablet_bootstrap.cc:492] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: No bootstrap required, opened a new log
I20260926 01:08:44.862599 20029 ts_tablet_manager.cc:1397] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:08:44.862637 20027 ts_tablet_manager.cc:1397] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 01:08:44.864114 20028 raft_consensus.cc:348] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 } } peers { permanent_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46659 } } peers { permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 } }
I20260926 01:08:44.864326 20028 raft_consensus.cc:374] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:44.864363 20028 raft_consensus.cc:729] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7f1ed7a4b444f9b2b5e5cd310d8412, State: Initialized, Role: FOLLOWER
I20260926 01:08:44.864307 20029 raft_consensus.cc:348] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 } } peers { permanent_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46659 } } peers { permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 } }
I20260926 01:08:44.864486 20029 raft_consensus.cc:374] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:44.864528 20029 raft_consensus.cc:729] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a033f5b4a6dd43e1a73888f44a9d3a89, State: Initialized, Role: FOLLOWER
I20260926 01:08:44.864521 20028 consensus_queue.cc:260] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 } } peers { permanent_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46659 } } peers { permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 } }
I20260926 01:08:44.864641 20029 consensus_queue.cc:260] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 } } peers { permanent_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46659 } } peers { permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 } }
I20260926 01:08:44.864612 20027 raft_consensus.cc:348] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 } } peers { permanent_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46659 } } peers { permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 } }
I20260926 01:08:44.864779 20027 raft_consensus.cc:374] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:44.864820 20027 raft_consensus.cc:729] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607f3fbf7b2041c1b0b4a216232b2ba8, State: Initialized, Role: FOLLOWER
I20260926 01:08:44.864866 20028 ts_tablet_manager.cc:1428] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:44.864935 20029 ts_tablet_manager.cc:1428] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:08:44.864929 20027 consensus_queue.cc:260] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 } } peers { permanent_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46659 } } peers { permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 } }
I20260926 01:08:44.865015 20010 heartbeater.cc:503] Master 127.0.98.62:44379 was elected leader, sending a full tablet report...
I20260926 01:08:44.865200 20027 ts_tablet_manager.cc:1428] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:44.865224 19876 heartbeater.cc:503] Master 127.0.98.62:44379 was elected leader, sending a full tablet report...
I20260926 01:08:44.865500 19742 heartbeater.cc:503] Master 127.0.98.62:44379 was elected leader, sending a full tablet report...
W20260926 01:08:44.894186 19877 tablet.cc:2369] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:08:44.939479 20035 raft_consensus.cc:482] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:08:44.939613 20035 raft_consensus.cc:504] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 } } peers { permanent_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46659 } } peers { permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 } }
I20260926 01:08:44.940013 20035 leader_election.cc:288] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a033f5b4a6dd43e1a73888f44a9d3a89 (127.0.98.1:46327), ab7f1ed7a4b444f9b2b5e5cd310d8412 (127.0.98.3:34751)
I20260926 01:08:44.942775 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.940285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57277 (local address 127.0.98.1:46327)
0926 01:08:44.940331 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.940333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.940344 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:08:44.940376 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.940377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.940388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.940453 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:44.940611 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.941080 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.942018 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.942447 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.942513 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.942525 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.942667 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.942705 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:08:44.942814 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.940191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46327 (local address 127.0.98.2:57277)
0926 01:08:44.940236 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:44.940249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:44.940319 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:44.940473 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:44.940474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:44.940476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:44.940550 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.940552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.941156 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.941157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.941924 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.941926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.942519 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.942520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.942603 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.942606 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:44.942652 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:08:44.942697 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 01:08:44.942824 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.940334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35869 (local address 127.0.98.3:34751)
0926 01:08:44.940414 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.940416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.940459 (+ 43us) server_negotiation.cc:407] Connection header received
0926 01:08:44.940516 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.940517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.940526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.940572 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:44.940711 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.941156 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.942080 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.942510 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.942572 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.942583 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.942717 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.942755 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 01:08:44.942881 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.940239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34751 (local address 127.0.98.2:35869)
0926 01:08:44.940411 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:44.940420 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:44.940469 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:44.940588 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:44.940589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:44.940590 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:44.940655 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.940656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.941235 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.941236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.941993 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:44.941995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.942583 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.942584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:44.942657 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.942660 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:44.942706 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:08:44.942745 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":121,"threads_started":1}
I20260926 01:08:44.943157 19696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7570178d66649db88e718cfc7087cab" candidate_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" is_pre_election: true
I20260926 01:08:44.943228 19964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7570178d66649db88e718cfc7087cab" candidate_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" is_pre_election: true
I20260926 01:08:44.943313 19696 raft_consensus.cc:2454] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607f3fbf7b2041c1b0b4a216232b2ba8 in term 0.
I20260926 01:08:44.943387 19964 raft_consensus.cc:2454] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607f3fbf7b2041c1b0b4a216232b2ba8 in term 0.
I20260926 01:08:44.943575 19765 leader_election.cc:302] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607f3fbf7b2041c1b0b4a216232b2ba8, a033f5b4a6dd43e1a73888f44a9d3a89; no voters:
I20260926 01:08:44.943713 20035 raft_consensus.cc:2781] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:08:44.943775 20035 raft_consensus.cc:482] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:08:44.943792 20035 raft_consensus.cc:3037] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:44.944439 20035 raft_consensus.cc:504] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 } } peers { permanent_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46659 } } peers { permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 } }
I20260926 01:08:44.944568 20035 leader_election.cc:288] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [CANDIDATE]: Term 1 election: Requested vote from peers a033f5b4a6dd43e1a73888f44a9d3a89 (127.0.98.1:46327), ab7f1ed7a4b444f9b2b5e5cd310d8412 (127.0.98.3:34751)
I20260926 01:08:44.944761 19964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7570178d66649db88e718cfc7087cab" candidate_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412"
I20260926 01:08:44.944775 19696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7570178d66649db88e718cfc7087cab" candidate_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89"
I20260926 01:08:44.944835 19964 raft_consensus.cc:3037] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:44.944854 19696 raft_consensus.cc:3037] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:44.945544 19964 raft_consensus.cc:2454] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607f3fbf7b2041c1b0b4a216232b2ba8 in term 1.
I20260926 01:08:44.945600 19696 raft_consensus.cc:2454] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607f3fbf7b2041c1b0b4a216232b2ba8 in term 1.
I20260926 01:08:44.945750 19764 leader_election.cc:302] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607f3fbf7b2041c1b0b4a216232b2ba8, ab7f1ed7a4b444f9b2b5e5cd310d8412; no voters:
I20260926 01:08:44.945870 20035 raft_consensus.cc:2781] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:44.946019 20035 raft_consensus.cc:686] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 1 LEADER]: Becoming Leader. State: Replica: 607f3fbf7b2041c1b0b4a216232b2ba8, State: Running, Role: LEADER
I20260926 01:08:44.946110 20035 consensus_queue.cc:237] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 } } peers { permanent_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46659 } } peers { permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 } }
I20260926 01:08:44.946959 19551 catalog_manager.cc:5495] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 607f3fbf7b2041c1b0b4a216232b2ba8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "607f3fbf7b2041c1b0b4a216232b2ba8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 } health_report { overall_health: UNKNOWN } } }
I20260926 01:08:44.974471 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.972039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.0.98.2:46659)
0926 01:08:44.972093 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:44.972095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:44.972103 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:08:44.972141 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:44.972142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:44.972163 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:44.972222 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:08:44.972387 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.972878 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.973725 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:44.973927 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:44.973990 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:44.974160 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:08:44.974265 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:08:44.974320 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:44.974335 (+ 15us) server_negotiation.cc:299] Negotiation successful
0926 01:08:44.974353 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:08:44.976316 19696 raft_consensus.cc:1261] T c7570178d66649db88e718cfc7087cab P a033f5b4a6dd43e1a73888f44a9d3a89 [term 1 FOLLOWER]: Refusing update from remote peer 607f3fbf7b2041c1b0b4a216232b2ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:44.976316 19964 raft_consensus.cc:1261] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412 [term 1 FOLLOWER]: Refusing update from remote peer 607f3fbf7b2041c1b0b4a216232b2ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:44.976697 20037 consensus_queue.cc:1035] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7f1ed7a4b444f9b2b5e5cd310d8412" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:44.976840 20035 consensus_queue.cc:1035] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a033f5b4a6dd43e1a73888f44a9d3a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:44.979070 20040 mvcc.cc:205] Tried to move back new op lower bound from 7333416652701048832 to 7333416652580556800. Current Snapshot: MvccSnapshot[applied={T|T < 7333416652701048832}]
I20260926 01:08:44.979789 20041 mvcc.cc:205] Tried to move back new op lower bound from 7333416652701048832 to 7333416652580556800. Current Snapshot: MvccSnapshot[applied={T|T < 7333416652701048832 or (T in {7333416652701048832})}]
I20260926 01:08:44.979854 20042 mvcc.cc:205] Tried to move back new op lower bound from 7333416652701048832 to 7333416652580556800. Current Snapshot: MvccSnapshot[applied={T|T < 7333416652701048832}]
I20260926 01:08:44.981825 19551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:44.981928 19551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:44.982694 19551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c159ec23f824a15a305ffe908cd1574: Sending DeleteTablet for 3 replicas of tablet c7570178d66649db88e718cfc7087cab
I20260926 01:08:44.983017 19676 tablet_service.cc:1514] Processing DeleteTablet for tablet c7570178d66649db88e718cfc7087cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56456
I20260926 01:08:44.983088 19944 tablet_service.cc:1514] Processing DeleteTablet for tablet c7570178d66649db88e718cfc7087cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54622
I20260926 01:08:44.983076 19810 tablet_service.cc:1514] Processing DeleteTablet for tablet c7570178d66649db88e718cfc7087cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54278
I20260926 01:08:44.983294 20050 tablet_replica.cc:330] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: stopping tablet replica
I20260926 01:08:44.983397 20050 raft_consensus.cc:2229] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:08:44.983518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 19613
I20260926 01:08:44.983563 20050 raft_consensus.cc:2258] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:44.984310 20051 tablet_replica.cc:330] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: stopping tablet replica
I20260926 01:08:44.984407 20051 raft_consensus.cc:2229] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:08:44.984500 20051 raft_consensus.cc:2258] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:44.984958 20051 ts_tablet_manager.cc:1905] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:44.985589 20050 ts_tablet_manager.cc:1905] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:44.986723 20051 ts_tablet_manager.cc:1918] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:44.986801 20051 log.cc:1196] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/c7570178d66649db88e718cfc7087cab
I20260926 01:08:44.987046 20051 ts_tablet_manager.cc:1939] T c7570178d66649db88e718cfc7087cab P ab7f1ed7a4b444f9b2b5e5cd310d8412: Deleting consensus metadata
I20260926 01:08:44.987188 20050 ts_tablet_manager.cc:1918] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:44.987259 20050 log.cc:1196] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/c7570178d66649db88e718cfc7087cab
I20260926 01:08:44.987473 20050 ts_tablet_manager.cc:1939] T c7570178d66649db88e718cfc7087cab P 607f3fbf7b2041c1b0b4a216232b2ba8: Deleting consensus metadata
I20260926 01:08:44.987622 19536 catalog_manager.cc:4841] TS ab7f1ed7a4b444f9b2b5e5cd310d8412 (127.0.98.3:34751): tablet c7570178d66649db88e718cfc7087cab (table test-table [id=cc77781d3c2e4d06b2da3a16f994f484]) successfully deleted
I20260926 01:08:44.987856 19535 catalog_manager.cc:4841] TS 607f3fbf7b2041c1b0b4a216232b2ba8 (127.0.98.2:46659): tablet c7570178d66649db88e718cfc7087cab (table test-table [id=cc77781d3c2e4d06b2da3a16f994f484]) successfully deleted
W20260926 01:08:44.994462 19537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46327 (error 108)
I20260926 01:08:44.994532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 19747
I20260926 01:08:44.996373 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:44.996227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46327 (local address 127.0.0.1:56472)
0926 01:08:44.996293 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:44.996337 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 01:08:44.996523 19537 catalog_manager.cc:4584] TS a033f5b4a6dd43e1a73888f44a9d3a89 (127.0.98.1:46327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7570178d66649db88e718cfc7087cab: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46327: connect: Connection refused (error 111)
I20260926 01:08:44.998787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 19881
I20260926 01:08:45.003167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 19519
2026-09-26T01:08:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1930 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T01:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:45Z Disabled control of system clock
I20260926 01:08:45.029639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42321
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:08:45.113476 20056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:45.113725 20056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:45.113747 20056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:45.115190 20056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:45.115257 20056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:45.115268 20056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:45.115281 20056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:45.115293 20056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:45.117012 20056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42321
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:45.117292 20056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:45.117612 20056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:45.122587 20064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:45.122601 20062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:45.122589 20061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:45.122808 20056 server_base.cc:1034] running on GCE node
I20260926 01:08:45.123126 20056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:45.123411 20056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:45.124688 20056 hybrid_clock.cc:648] HybridClock initialized: now 1790384925124664 us; error 41 us; skew 500 ppm
I20260926 01:08:45.126224 20056 webserver.cc:460] Webserver started at http://127.0.98.62:38021/ using document root <none> and password file <none>
I20260926 01:08:45.126487 20056 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:45.126540 20056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:45.126680 20056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:45.127770 20056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "1c287331227a45eb8b3b7fe5cdc8fd24"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "bec69a19181a904df8e2c8d57910b43c"
server_key_iv: "486d576fc0ccb208b84b368eff988aa8"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.128088 20056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c287331227a45eb8b3b7fe5cdc8fd24"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "bec69a19181a904df8e2c8d57910b43c"
server_key_iv: "486d576fc0ccb208b84b368eff988aa8"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.129380 20056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:08:45.130446 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:45.130609 20056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:08:45.130807 20056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "1c287331227a45eb8b3b7fe5cdc8fd24"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "bec69a19181a904df8e2c8d57910b43c"
server_key_iv: "486d576fc0ccb208b84b368eff988aa8"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.130916 20056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:45.148597 20056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:45.148914 20056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:45.149045 20056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:45.152930 20056 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42321
I20260926 01:08:45.152966 20122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42321 every 8 connection(s)
I20260926 01:08:45.153316 20056 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:45.154081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 20056
I20260926 01:08:45.154178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:45.154284 20123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:45.154376 392 external_mini_cluster.cc:1428] Setting key 94ecb0333230ba67d2c8e2ff533a9e16
I20260926 01:08:45.158002 20123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24: Bootstrap starting.
I20260926 01:08:45.158701 20123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:45.158993 20123 log.cc:824] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:45.159857 20123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24: No bootstrap required, opened a new log
I20260926 01:08:45.160615 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.154965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.0.98.62:42321)
0926 01:08:45.155151 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.155161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.155195 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:08:45.155703 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.155713 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.155796 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.155954 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:45.156191 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.157519 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.158374 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.159001 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.159137 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.159873 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:45.159878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:45.159879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:45.160084 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:45.160128 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:45.160144 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:45.160211 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.160299 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.160383 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260926 01:08:45.162047 20123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c287331227a45eb8b3b7fe5cdc8fd24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } }
I20260926 01:08:45.162187 20123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:45.162209 20123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c287331227a45eb8b3b7fe5cdc8fd24, State: Initialized, Role: FOLLOWER
I20260926 01:08:45.162328 20123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c287331227a45eb8b3b7fe5cdc8fd24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } }
I20260926 01:08:45.162433 20123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:45.162464 20123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:45.162513 20123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:45.163089 20123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c287331227a45eb8b3b7fe5cdc8fd24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } }
I20260926 01:08:45.163180 20123 leader_election.cc:302] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c287331227a45eb8b3b7fe5cdc8fd24; no voters:
I20260926 01:08:45.163352 20123 leader_election.cc:288] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:45.163424 20128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:45.163590 20128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [term 1 LEADER]: Becoming Leader. State: Replica: 1c287331227a45eb8b3b7fe5cdc8fd24, State: Running, Role: LEADER
I20260926 01:08:45.163610 20123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:45.163681 20128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c287331227a45eb8b3b7fe5cdc8fd24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } }
I20260926 01:08:45.164085 20129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c287331227a45eb8b3b7fe5cdc8fd24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c287331227a45eb8b3b7fe5cdc8fd24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } } }
I20260926 01:08:45.164184 20129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:45.164067 20130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c287331227a45eb8b3b7fe5cdc8fd24. Latest consensus state: current_term: 1 leader_uuid: "1c287331227a45eb8b3b7fe5cdc8fd24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c287331227a45eb8b3b7fe5cdc8fd24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42321 } } }
I20260926 01:08:45.164374 20130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:45.164812 20137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:45.165269 20137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:45.166795 20137 catalog_manager.cc:1347] Generated new cluster ID: b64755f3984849cea97b4ba9c5c8a99d
I20260926 01:08:45.166858 20137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:45.174012 20137 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:45.174823 20137 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:45.181622 20137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24: Generated new TSK 0
I20260926 01:08:45.181849 20137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:45.186795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--builtin_ntp_servers=127.0.98.20:41469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:08:45.271031 20147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:45.271270 20147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:45.271306 20147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:45.272632 20147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:45.272687 20147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:45.272727 20147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 01:08:45.274498 20147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:45.274873 20147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:45.275198 20147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:45.280982 20152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:45.280970 20155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:45.280966 20153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:45.281565 20147 server_base.cc:1034] running on GCE node
I20260926 01:08:45.281795 20147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:45.282120 20147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:45.283296 20147 hybrid_clock.cc:648] HybridClock initialized: now 1790384925283279 us; error 34 us; skew 500 ppm
I20260926 01:08:45.284832 20147 webserver.cc:460] Webserver started at http://127.0.98.1:46191/ using document root <none> and password file <none>
I20260926 01:08:45.285089 20147 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:45.285148 20147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:45.285274 20147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:45.286360 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/data/instance:
uuid: "50b6a8c83ee84660931ac64a39ef2598"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "beb5f327aeac71759ff8f6cad2b945ae"
server_key_iv: "bd46544ab0658ff68ae3c5900a047b6e"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.286755 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/wal/instance:
uuid: "50b6a8c83ee84660931ac64a39ef2598"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "beb5f327aeac71759ff8f6cad2b945ae"
server_key_iv: "bd46544ab0658ff68ae3c5900a047b6e"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.288053 20147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:08:45.288779 20161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:45.288956 20147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:08:45.289059 20147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/wal
uuid: "50b6a8c83ee84660931ac64a39ef2598"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "beb5f327aeac71759ff8f6cad2b945ae"
server_key_iv: "bd46544ab0658ff68ae3c5900a047b6e"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.289137 20147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:45.300674 20147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:45.301055 20147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:45.301200 20147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:45.301441 20147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:45.301802 20147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:45.301843 20147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:45.301875 20147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:45.301895 20147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:45.307964 20147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33547
I20260926 01:08:45.308056 20274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33547 every 8 connection(s)
I20260926 01:08:45.308439 20147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/data/info.pb
I20260926 01:08:45.312286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 20147
I20260926 01:08:45.312399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-0/wal/instance
I20260926 01:08:45.312623 392 external_mini_cluster.cc:1428] Setting key 949fd90d84865b5fb5d2dce0f8936f84
I20260926 01:08:45.313599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--builtin_ntp_servers=127.0.98.20:41469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 01:08:45.314056 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.309341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55377 (local address 127.0.98.62:42321)
0926 01:08:45.309397 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.309399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.309609 (+ 210us) server_negotiation.cc:407] Connection header received
0926 01:08:45.310153 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.310155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.310168 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.310223 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:45.310962 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.311459 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.312497 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.312783 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.312862 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.313749 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:45.313750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:45.313751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:45.313755 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:45.313772 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:45.313776 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:45.313829 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.313943 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.313974 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 01:08:45.314311 20277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.309294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42321 (local address 127.0.98.1:55377)
0926 01:08:45.309490 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:45.309526 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:45.310057 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:45.310308 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:45.310312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:45.310320 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:45.310869 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.310875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.311553 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.311554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.312390 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.312392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.312885 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.312886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.313035 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.313589 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:08:45.313589 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:45.313619 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:08:45.313644 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:45.313837 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:45.313841 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:45.313845 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:45.313938 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 01:08:45.314030 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":116,"threads_started":1}
I20260926 01:08:45.314867 20275 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42321
I20260926 01:08:45.314991 20275 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:45.315205 20275 heartbeater.cc:511] Master 127.0.98.62:42321 requested a full tablet report, sending...
I20260926 01:08:45.315896 20087 ts_manager.cc:194] Registered new tserver with Master: 50b6a8c83ee84660931ac64a39ef2598 (127.0.98.1:33547)
I20260926 01:08:45.316799 20087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55377
W20260926 01:08:45.398486 20278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:45.398702 20278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:45.398725 20278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:45.399895 20278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:45.399937 20278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:45.399961 20278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 01:08:45.401284 20278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:45.401496 20278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:45.401755 20278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:45.407030 20283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:45.407051 20284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:45.407030 20286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:45.407050 20278 server_base.cc:1034] running on GCE node
I20260926 01:08:45.407431 20278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:45.407670 20278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:45.408847 20278 hybrid_clock.cc:648] HybridClock initialized: now 1790384925408825 us; error 32 us; skew 500 ppm
I20260926 01:08:45.410177 20278 webserver.cc:460] Webserver started at http://127.0.98.2:43551/ using document root <none> and password file <none>
I20260926 01:08:45.410465 20278 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:45.410521 20278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:45.410646 20278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:45.411752 20278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/data/instance:
uuid: "d838d8c7ca51458db9f1576f3f5a8e9d"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "94777076331f845a55b9a57fa32b627f"
server_key_iv: "6aa64cbd0393fc0331e9abe702425495"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.412091 20278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d838d8c7ca51458db9f1576f3f5a8e9d"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "94777076331f845a55b9a57fa32b627f"
server_key_iv: "6aa64cbd0393fc0331e9abe702425495"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.413331 20278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:08:45.414178 20292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:45.414422 20278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:08:45.414520 20278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/wal
uuid: "d838d8c7ca51458db9f1576f3f5a8e9d"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "94777076331f845a55b9a57fa32b627f"
server_key_iv: "6aa64cbd0393fc0331e9abe702425495"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.414588 20278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:45.422343 20278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:45.422686 20278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:45.422821 20278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:45.423055 20278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:45.423365 20278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:45.423401 20278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:45.423431 20278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:45.423455 20278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:45.428943 20278 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35641
I20260926 01:08:45.428988 20405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35641 every 8 connection(s)
I20260926 01:08:45.429308 20278 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/data/info.pb
I20260926 01:08:45.434996 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.429996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40105 (local address 127.0.98.62:42321)
0926 01:08:45.430049 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.430051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.430406 (+ 355us) server_negotiation.cc:407] Connection header received
0926 01:08:45.431003 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.431004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.431018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.431091 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:45.431782 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.432280 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.433280 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.433652 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.433734 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.434710 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:45.434712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:45.434712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:45.434718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:45.434735 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:45.434739 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:45.434786 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.434869 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.434901 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:08:45.435221 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.430034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42321 (local address 127.0.98.2:40105)
0926 01:08:45.430251 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:45.430288 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:45.430827 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:45.431168 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:45.431172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:45.431183 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:45.431689 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.431694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.432376 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.432377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.433188 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.433191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.433764 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.433765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.433925 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.434561 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:08:45.434561 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:45.434592 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:08:45.434620 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:45.434792 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:45.434798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:45.434801 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:45.434867 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 01:08:45.434992 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
I20260926 01:08:45.435675 20406 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42321
I20260926 01:08:45.435758 20406 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:45.435937 20406 heartbeater.cc:511] Master 127.0.98.62:42321 requested a full tablet report, sending...
I20260926 01:08:45.436343 20087 ts_manager.cc:194] Registered new tserver with Master: d838d8c7ca51458db9f1576f3f5a8e9d (127.0.98.2:35641)
I20260926 01:08:45.436725 20087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40105
I20260926 01:08:45.439144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 20278
I20260926 01:08:45.439230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/wal/instance
I20260926 01:08:45.439433 392 external_mini_cluster.cc:1428] Setting key be5d5a5c1935ae707f938f5589014855
I20260926 01:08:45.440451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--builtin_ntp_servers=127.0.98.20:41469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 01:08:45.523257 20409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:45.523488 20409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:45.523514 20409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:45.524777 20409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:45.524832 20409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:45.524868 20409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 01:08:45.526373 20409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:45.526672 20409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:45.526966 20409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:45.531970 20414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:45.531965 20417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:45.532042 20409 server_base.cc:1034] running on GCE node
W20260926 01:08:45.531965 20415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:45.532405 20409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:45.532646 20409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:45.533830 20409 hybrid_clock.cc:648] HybridClock initialized: now 1790384925533796 us; error 45 us; skew 500 ppm
I20260926 01:08:45.535231 20409 webserver.cc:460] Webserver started at http://127.0.98.3:44251/ using document root <none> and password file <none>
I20260926 01:08:45.535499 20409 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:45.535548 20409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:45.535671 20409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:45.536738 20409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/data/instance:
uuid: "d069aa0cc7294d23b4e85c284630e1c3"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "3019aa5c801cf3ab814acc26f7aa7031"
server_key_iv: "cdd8bb7dd4157e6e325a0d2afdb79fba"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.537062 20409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d069aa0cc7294d23b4e85c284630e1c3"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "3019aa5c801cf3ab814acc26f7aa7031"
server_key_iv: "cdd8bb7dd4157e6e325a0d2afdb79fba"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.538319 20409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:45.539135 20423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:45.539325 20409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:08:45.539409 20409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/wal
uuid: "d069aa0cc7294d23b4e85c284630e1c3"
format_stamp: "Formatted at 2026-09-26 01:08:45 on dist-test-slave-l73v"
server_key: "3019aa5c801cf3ab814acc26f7aa7031"
server_key_iv: "cdd8bb7dd4157e6e325a0d2afdb79fba"
server_key_version: "encryptionkey@0"
I20260926 01:08:45.539471 20409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:45.570094 20409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:45.570457 20409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:45.570586 20409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:45.570811 20409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 01:08:45.571126 20409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:08:45.571159 20409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:45.571188 20409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 01:08:45.571210 20409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:45.576583 20409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37487
I20260926 01:08:45.576639 20536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37487 every 8 connection(s)
I20260926 01:08:45.576957 20409 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/data/info.pb
I20260926 01:08:45.582402 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.577694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39357 (local address 127.0.98.62:42321)
0926 01:08:45.577759 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.577762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.578019 (+ 257us) server_negotiation.cc:407] Connection header received
0926 01:08:45.578527 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.578528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.578539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.578602 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:45.579277 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.579796 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.580913 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.581108 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.581174 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.582099 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:45.582100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:45.582100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:45.582106 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:45.582122 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:45.582126 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:45.582169 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.582287 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.582315 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:08:45.582654 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.577635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42321 (local address 127.0.98.3:39357)
0926 01:08:45.577905 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:45.577937 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:45.578432 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:45.578659 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:45.578662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:45.578671 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:08:45.579167 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.579172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.579933 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.579934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.580780 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.580784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.581217 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.581218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.581392 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.581954 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 01:08:45.581955 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:08:45.581986 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 01:08:45.582011 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:08:45.582205 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:08:45.582212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:08:45.582215 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 01:08:45.582287 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 01:08:45.582423 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":73,"threads_started":1}
I20260926 01:08:45.583169 20537 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42321
I20260926 01:08:45.583261 20537 heartbeater.cc:464] Registering TS with master...
I20260926 01:08:45.583421 20537 heartbeater.cc:511] Master 127.0.98.62:42321 requested a full tablet report, sending...
I20260926 01:08:45.583804 20087 ts_manager.cc:194] Registered new tserver with Master: d069aa0cc7294d23b4e85c284630e1c3 (127.0.98.3:37487)
I20260926 01:08:45.584194 20087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39357
I20260926 01:08:45.585888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 20409
I20260926 01:08:45.585971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/wal/instance
I20260926 01:08:45.586110 392 external_mini_cluster.cc:1428] Setting key 1a338076aa36d981ab60e60cdd805a1b
I20260926 01:08:45.587074 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 01:08:45.591197 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.588365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.0.98.62:42321)
0926 01:08:45.588449 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.588451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.588513 (+ 62us) server_negotiation.cc:407] Connection header received
0926 01:08:45.588576 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.588577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.588587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.588639 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:45.588783 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.589275 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.590016 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.590216 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.590271 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.590965 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:45.590967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:45.590967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:45.590972 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:45.590987 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:45.590991 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:45.591038 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.591085 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.591112 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 01:08:45.592660 20087 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:08:45.594265 20087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:08:45.600549 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.596726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59586 (local address 127.0.98.3:37487)
0926 01:08:45.596909 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.596913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.596922 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:08:45.596959 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.596961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.596976 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.597212 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:45.597276 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.598045 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.599476 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.600140 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.600208 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.600309 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.600416 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.600453 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":121,"threads_started":1}
I20260926 01:08:45.600543 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.596446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37487 (local address 127.0.0.1:59586)
0926 01:08:45.596654 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:45.596695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:45.596760 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:45.597073 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:45.597074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:45.597076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:45.597139 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.597141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.598120 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.598121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.599387 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.599390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.600227 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.600228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.600312 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.600326 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 01:08:45.600402 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 01:08:45.600456 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":5,"thread_start_us":125,"threads_started":1}
I20260926 01:08:45.600792 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.596446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33547 (local address 127.0.0.1:55416)
0926 01:08:45.596858 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:45.596865 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:45.596900 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:45.597395 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:45.597395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:45.597397 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:45.597430 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.597432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.598661 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.598662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.599635 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.599639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.600609 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.600610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.600659 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.600663 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:45.600702 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 01:08:45.600737 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":22,"thread_start_us":118,"threads_started":1}
I20260926 01:08:45.600921 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.596699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.0.98.2:35641)
0926 01:08:45.596895 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.596900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.596921 (+ 21us) server_negotiation.cc:407] Connection header received
0926 01:08:45.596959 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.596961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.596975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.597043 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:45.597219 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.598338 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.599540 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.600264 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.600336 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.600423 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.600780 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.600822 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20260926 01:08:45.600693 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.596446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35641 (local address 127.0.0.1:52932)
0926 01:08:45.596654 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:45.596695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:45.596760 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:45.597056 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:45.597059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:45.597060 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:45.597138 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.597140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.598449 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.598451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.599479 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.599481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.600460 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.600461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.600524 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.600528 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:45.600570 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 01:08:45.600613 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260926 01:08:45.601084 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.596965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.0.98.1:33547)
0926 01:08:45.597198 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.597202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.597211 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:08:45.597246 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.597248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.597263 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.597331 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:45.597513 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.598589 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.599735 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.600445 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.600504 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.600580 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.600967 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.601010 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260926 01:08:45.601603 20471 tablet_service.cc:1467] Processing CreateTablet for tablet 751531c40af7467088f56a6cfdc41400 (DEFAULT_TABLE table=test-table [id=041f0c1dc8954e8baa28a5f6a39130b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:45.601593 20340 tablet_service.cc:1467] Processing CreateTablet for tablet 751531c40af7467088f56a6cfdc41400 (DEFAULT_TABLE table=test-table [id=041f0c1dc8954e8baa28a5f6a39130b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:45.601912 20471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751531c40af7467088f56a6cfdc41400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:45.601914 20340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751531c40af7467088f56a6cfdc41400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:45.604635 20551 tablet_bootstrap.cc:492] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: Bootstrap starting.
I20260926 01:08:45.605337 20551 tablet_bootstrap.cc:654] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:45.605190 20209 tablet_service.cc:1467] Processing CreateTablet for tablet 751531c40af7467088f56a6cfdc41400 (DEFAULT_TABLE table=test-table [id=041f0c1dc8954e8baa28a5f6a39130b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:08:45.605569 20209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751531c40af7467088f56a6cfdc41400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:45.605742 20551 log.cc:824] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:45.606096 20552 tablet_bootstrap.cc:492] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: Bootstrap starting.
I20260926 01:08:45.606534 20551 tablet_bootstrap.cc:492] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: No bootstrap required, opened a new log
I20260926 01:08:45.606600 20551 ts_tablet_manager.cc:1397] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 01:08:45.606768 20552 tablet_bootstrap.cc:654] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:45.607076 20552 log.cc:824] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:45.607960 20552 tablet_bootstrap.cc:492] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: No bootstrap required, opened a new log
I20260926 01:08:45.608040 20552 ts_tablet_manager.cc:1397] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 01:08:45.609125 20551 raft_consensus.cc:348] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d069aa0cc7294d23b4e85c284630e1c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37487 } } peers { permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 } } peers { permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 } }
I20260926 01:08:45.609310 20551 raft_consensus.cc:374] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:45.609368 20551 raft_consensus.cc:729] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d838d8c7ca51458db9f1576f3f5a8e9d, State: Initialized, Role: FOLLOWER
I20260926 01:08:45.609489 20551 consensus_queue.cc:260] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d069aa0cc7294d23b4e85c284630e1c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37487 } } peers { permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 } } peers { permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 } }
I20260926 01:08:45.609709 20555 tablet_bootstrap.cc:492] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598: Bootstrap starting.
I20260926 01:08:45.609738 20551 ts_tablet_manager.cc:1428] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:45.609889 20406 heartbeater.cc:503] Master 127.0.98.62:42321 was elected leader, sending a full tablet report...
I20260926 01:08:45.610411 20555 tablet_bootstrap.cc:654] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:45.610481 20552 raft_consensus.cc:348] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d069aa0cc7294d23b4e85c284630e1c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37487 } } peers { permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 } } peers { permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 } }
I20260926 01:08:45.610630 20552 raft_consensus.cc:374] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:45.610663 20552 raft_consensus.cc:729] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d069aa0cc7294d23b4e85c284630e1c3, State: Initialized, Role: FOLLOWER
I20260926 01:08:45.610702 20555 log.cc:824] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:45.610776 20552 consensus_queue.cc:260] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d069aa0cc7294d23b4e85c284630e1c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37487 } } peers { permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 } } peers { permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 } }
I20260926 01:08:45.611016 20552 ts_tablet_manager.cc:1428] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 01:08:45.611089 20537 heartbeater.cc:503] Master 127.0.98.62:42321 was elected leader, sending a full tablet report...
I20260926 01:08:45.611500 20555 tablet_bootstrap.cc:492] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598: No bootstrap required, opened a new log
I20260926 01:08:45.611570 20555 ts_tablet_manager.cc:1397] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 01:08:45.611996 20558 raft_consensus.cc:482] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:08:45.612121 20558 raft_consensus.cc:504] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d069aa0cc7294d23b4e85c284630e1c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37487 } } peers { permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 } } peers { permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 } }
I20260926 01:08:45.612406 20558 leader_election.cc:288] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d838d8c7ca51458db9f1576f3f5a8e9d (127.0.98.2:35641), 50b6a8c83ee84660931ac64a39ef2598 (127.0.98.1:33547)
I20260926 01:08:45.614142 20555 raft_consensus.cc:348] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d069aa0cc7294d23b4e85c284630e1c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37487 } } peers { permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 } } peers { permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 } }
I20260926 01:08:45.614315 20555 raft_consensus.cc:374] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:45.614365 20555 raft_consensus.cc:729] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b6a8c83ee84660931ac64a39ef2598, State: Initialized, Role: FOLLOWER
I20260926 01:08:45.614521 20555 consensus_queue.cc:260] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d069aa0cc7294d23b4e85c284630e1c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37487 } } peers { permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 } } peers { permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 } }
I20260926 01:08:45.614794 20555 ts_tablet_manager.cc:1428] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 01:08:45.614820 20275 heartbeater.cc:503] Master 127.0.98.62:42321 was elected leader, sending a full tablet report...
I20260926 01:08:45.615792 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.612537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35641 (local address 127.0.98.3:53645)
0926 01:08:45.612622 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:45.612632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:45.612670 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:45.612797 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:45.612798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:45.612799 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:45.612863 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.612864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.613614 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.613615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.614728 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.614731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.615532 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.615533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.615617 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.615621 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 01:08:45.615671 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:08:45.615716 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 01:08:45.616009 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.612586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53645 (local address 127.0.98.2:35641)
0926 01:08:45.612626 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.612628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.612704 (+ 76us) server_negotiation.cc:407] Connection header received
0926 01:08:45.612728 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.612729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.612737 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.612783 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:45.612922 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.613518 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.614815 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.615393 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.615482 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.615498 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.615896 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.615946 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:08:45.616331 20360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751531c40af7467088f56a6cfdc41400" candidate_uuid: "d069aa0cc7294d23b4e85c284630e1c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" is_pre_election: true
I20260926 01:08:45.616468 20360 raft_consensus.cc:2454] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d069aa0cc7294d23b4e85c284630e1c3 in term 0.
I20260926 01:08:45.616691 20427 leader_election.cc:302] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d069aa0cc7294d23b4e85c284630e1c3, d838d8c7ca51458db9f1576f3f5a8e9d; no voters:
I20260926 01:08:45.616814 20558 raft_consensus.cc:2781] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:08:45.616863 20558 raft_consensus.cc:482] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:08:45.616882 20558 raft_consensus.cc:3037] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:45.617630 20558 raft_consensus.cc:504] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d069aa0cc7294d23b4e85c284630e1c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37487 } } peers { permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 } } peers { permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 } }
I20260926 01:08:45.617849 20558 leader_election.cc:288] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [CANDIDATE]: Term 1 election: Requested vote from peers d838d8c7ca51458db9f1576f3f5a8e9d (127.0.98.2:35641), 50b6a8c83ee84660931ac64a39ef2598 (127.0.98.1:33547)
I20260926 01:08:45.617915 20360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751531c40af7467088f56a6cfdc41400" candidate_uuid: "d069aa0cc7294d23b4e85c284630e1c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d"
I20260926 01:08:45.618007 20360 raft_consensus.cc:3037] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:45.618366 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.612727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46077 (local address 127.0.98.1:33547)
0926 01:08:45.612774 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.612776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.615021 (+ 2245us) server_negotiation.cc:407] Connection header received
0926 01:08:45.615088 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.615089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.615101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.615159 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 01:08:45.615473 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.616155 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.617536 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.618055 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.618122 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.618133 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.618264 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.618302 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:08:45.618413 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.612547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33547 (local address 127.0.98.3:46077)
0926 01:08:45.614935 (+ 2388us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:45.614947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 01:08:45.615022 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:08:45.615294 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:08:45.615295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:08:45.615297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 01:08:45.615379 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.615382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.616568 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.616570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.617457 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:08:45.617460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.618127 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.618128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:08:45.618202 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.618207 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 01:08:45.618251 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 01:08:45.618292 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2367,"thread_start_us":88,"threads_started":1}
I20260926 01:08:45.618968 20360 raft_consensus.cc:2454] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d069aa0cc7294d23b4e85c284630e1c3 in term 1.
I20260926 01:08:45.619140 20427 leader_election.cc:302] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d069aa0cc7294d23b4e85c284630e1c3, d838d8c7ca51458db9f1576f3f5a8e9d; no voters:
I20260926 01:08:45.619230 20558 raft_consensus.cc:2781] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:45.619287 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751531c40af7467088f56a6cfdc41400" candidate_uuid: "d069aa0cc7294d23b4e85c284630e1c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6a8c83ee84660931ac64a39ef2598"
I20260926 01:08:45.619295 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751531c40af7467088f56a6cfdc41400" candidate_uuid: "d069aa0cc7294d23b4e85c284630e1c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6a8c83ee84660931ac64a39ef2598" is_pre_election: true
I20260926 01:08:45.619414 20228 raft_consensus.cc:3037] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:45.619629 20558 raft_consensus.cc:686] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 1 LEADER]: Becoming Leader. State: Replica: d069aa0cc7294d23b4e85c284630e1c3, State: Running, Role: LEADER
I20260926 01:08:45.619726 20558 consensus_queue.cc:237] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d069aa0cc7294d23b4e85c284630e1c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37487 } } peers { permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 } } peers { permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 } }
I20260926 01:08:45.620245 20228 raft_consensus.cc:2454] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d069aa0cc7294d23b4e85c284630e1c3 in term 1.
I20260926 01:08:45.620631 20087 catalog_manager.cc:5495] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d069aa0cc7294d23b4e85c284630e1c3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d069aa0cc7294d23b4e85c284630e1c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d069aa0cc7294d23b4e85c284630e1c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 } health_report { overall_health: UNKNOWN } } }
I20260926 01:08:45.632519 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.629299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.0.98.3:37487)
0926 01:08:45.629410 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:45.629412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:45.629420 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:08:45.629456 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:45.629457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:45.629475 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:45.629543 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:08:45.629767 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.630492 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.631493 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:45.631833 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:45.631903 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:45.632160 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:08:45.632270 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:08:45.632338 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:45.632418 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 01:08:45.632438 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 01:08:45.634539 20360 raft_consensus.cc:1261] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d [term 1 FOLLOWER]: Refusing update from remote peer d069aa0cc7294d23b4e85c284630e1c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:45.634524 20228 raft_consensus.cc:1261] T 751531c40af7467088f56a6cfdc41400 P 50b6a8c83ee84660931ac64a39ef2598 [term 1 FOLLOWER]: Refusing update from remote peer d069aa0cc7294d23b4e85c284630e1c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:08:45.634891 20561 consensus_queue.cc:1035] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6a8c83ee84660931ac64a39ef2598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:45.635041 20558 consensus_queue.cc:1035] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d838d8c7ca51458db9f1576f3f5a8e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:08:45.638317 20565 mvcc.cc:205] Tried to move back new op lower bound from 7333416655395917824 to 7333416655339388928. Current Snapshot: MvccSnapshot[applied={T|T < 7333416655395917824}]
I20260926 01:08:45.639040 20566 mvcc.cc:205] Tried to move back new op lower bound from 7333416655395917824 to 7333416655339388928. Current Snapshot: MvccSnapshot[applied={T|T < 7333416655395917824 or (T in {7333416655395917824})}]
I20260926 01:08:45.639985 20087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:45.640089 20087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 01:08:45.640841 20087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c287331227a45eb8b3b7fe5cdc8fd24: Sending DeleteTablet for 3 replicas of tablet 751531c40af7467088f56a6cfdc41400
I20260926 01:08:45.641121 20471 tablet_service.cc:1514] Processing DeleteTablet for tablet 751531c40af7467088f56a6cfdc41400 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:45 UTC) from {username='slave'} at 127.0.0.1:59586
I20260926 01:08:45.641201 20340 tablet_service.cc:1514] Processing DeleteTablet for tablet 751531c40af7467088f56a6cfdc41400 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:45 UTC) from {username='slave'} at 127.0.0.1:52932
I20260926 01:08:45.641417 20209 tablet_service.cc:1514] Processing DeleteTablet for tablet 751531c40af7467088f56a6cfdc41400 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:08:45 UTC) from {username='slave'} at 127.0.0.1:55416
I20260926 01:08:45.641525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 20147
I20260926 01:08:45.641609 20574 tablet_replica.cc:330] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: stopping tablet replica
I20260926 01:08:45.643337 20574 raft_consensus.cc:2229] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:08:45.643573 20574 raft_consensus.cc:2258] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:45.643821 20573 tablet_replica.cc:330] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: stopping tablet replica
I20260926 01:08:45.643921 20573 raft_consensus.cc:2229] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:08:45.644012 20573 raft_consensus.cc:2258] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:08:45.644405 20574 ts_tablet_manager.cc:1905] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:45.644553 20573 ts_tablet_manager.cc:1905] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:08:45.646301 20574 ts_tablet_manager.cc:1918] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:45.646404 20574 log.cc:1196] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-1/wal/wals/751531c40af7467088f56a6cfdc41400
I20260926 01:08:45.646345 20573 ts_tablet_manager.cc:1918] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:08:45.646644 20573 log.cc:1196] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: Deleting WAL directory at /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790384680280236-392-0/minicluster-data/ts-2/wal/wals/751531c40af7467088f56a6cfdc41400
I20260926 01:08:45.646646 20574 ts_tablet_manager.cc:1939] T 751531c40af7467088f56a6cfdc41400 P d838d8c7ca51458db9f1576f3f5a8e9d: Deleting consensus metadata
I20260926 01:08:45.646960 20573 ts_tablet_manager.cc:1939] T 751531c40af7467088f56a6cfdc41400 P d069aa0cc7294d23b4e85c284630e1c3: Deleting consensus metadata
I20260926 01:08:45.647255 20074 catalog_manager.cc:4841] TS d838d8c7ca51458db9f1576f3f5a8e9d (127.0.98.2:35641): tablet 751531c40af7467088f56a6cfdc41400 (table test-table [id=041f0c1dc8954e8baa28a5f6a39130b8]) successfully deleted
I20260926 01:08:45.647372 20072 catalog_manager.cc:4841] TS d069aa0cc7294d23b4e85c284630e1c3 (127.0.98.3:37487): tablet 751531c40af7467088f56a6cfdc41400 (table test-table [id=041f0c1dc8954e8baa28a5f6a39130b8]) successfully deleted
W20260926 01:08:45.649096 20073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33547 (error 108)
I20260926 01:08:45.649597 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:45.649456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33547 (local address 127.0.0.1:55424)
0926 01:08:45.649512 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 01:08:45.649558 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 01:08:45.649597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 20278
W20260926 01:08:45.651082 20073 catalog_manager.cc:4584] TS 50b6a8c83ee84660931ac64a39ef2598 (127.0.98.1:33547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 751531c40af7467088f56a6cfdc41400: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33547: connect: Connection refused (error 111)
I20260926 01:08:45.653926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 20409
I20260926 01:08:45.660616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 20056
2026-09-26T01:08:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (656 ms)
[----------] 6 tests from EncryptionPolicyTest (7874 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:45 dist-test-slave-l73v krb5kdc[20579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:45 dist-test-slave-l73v krb5kdc[20579](info): set up 2 sockets
Sep 26 01:08:45 dist-test-slave-l73v krb5kdc[20579](info): commencing operation
krb5kdc: starting...
W20260926 01:08:46.622160 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:46 dist-test-slave-l73v krb5kdc[20579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:46.743423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:46.839581 20592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:46.839902 20592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:46.839942 20592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:46.841375 20592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:46.841436 20592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:46.841449 20592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:46.841461 20592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:46.841471 20592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:46.843127 20592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:46.843348 20592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:46.843537 20592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:46.848677 20600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:46.848691 20597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:46.848829 20592 server_base.cc:1034] running on GCE node
W20260926 01:08:46.848677 20598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:46.849166 20592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:46.849474 20592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:46.850669 20592 hybrid_clock.cc:648] HybridClock initialized: now 1790384926850656 us; error 33 us; skew 500 ppm
Sep 26 01:08:46 dist-test-slave-l73v krb5kdc[20579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:46.853299 20592 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:46.853842 20592 webserver.cc:460] Webserver started at http://127.0.0.1:41625/ using document root <none> and password file <none>
I20260926 01:08:46.854076 20592 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:46.854127 20592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:46.854259 20592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:46.855453 20592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "32c82d7aafb447b0b94672baf0141973"
format_stamp: "Formatted at 2026-09-26 01:08:46 on dist-test-slave-l73v"
server_key: "91a7906c0c26782da5d7a7377b960d23"
server_key_iv: "ea6d7c80a94a1c8ef6367c56b6cd9aba"
server_key_version: "encryptionkey@0"
I20260926 01:08:46.855788 20592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "32c82d7aafb447b0b94672baf0141973"
format_stamp: "Formatted at 2026-09-26 01:08:46 on dist-test-slave-l73v"
server_key: "91a7906c0c26782da5d7a7377b960d23"
server_key_iv: "ea6d7c80a94a1c8ef6367c56b6cd9aba"
server_key_version: "encryptionkey@0"
I20260926 01:08:46.857232 20592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:08:46.858093 20607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:46.858448 20592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:46.858551 20592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "32c82d7aafb447b0b94672baf0141973"
format_stamp: "Formatted at 2026-09-26 01:08:46 on dist-test-slave-l73v"
server_key: "91a7906c0c26782da5d7a7377b960d23"
server_key_iv: "ea6d7c80a94a1c8ef6367c56b6cd9aba"
server_key_version: "encryptionkey@0"
I20260926 01:08:46.858639 20592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:46.870864 20592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:46.873123 20592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:46.873247 20592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:46.877970 20592 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46111
I20260926 01:08:46.878059 20659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46111 every 8 connection(s)
I20260926 01:08:46.878408 20592 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:46.879326 20660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:46.881659 20660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973: Bootstrap starting.
I20260926 01:08:46.882220 20660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:46.882484 20660 log.cc:824] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:46.883200 20660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973: No bootstrap required, opened a new log
I20260926 01:08:46.884881 20660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c82d7aafb447b0b94672baf0141973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } }
I20260926 01:08:46.885042 20660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:46.885063 20660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c82d7aafb447b0b94672baf0141973, State: Initialized, Role: FOLLOWER
I20260926 01:08:46.885172 20660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c82d7aafb447b0b94672baf0141973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } }
I20260926 01:08:46.885229 20660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:46.885322 20660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:46.885432 20660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:46.886591 20660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c82d7aafb447b0b94672baf0141973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } }
I20260926 01:08:46.886730 20660 leader_election.cc:302] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c82d7aafb447b0b94672baf0141973; no voters:
I20260926 01:08:46.886919 20660 leader_election.cc:288] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:46.887006 20663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:46.887182 20663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [term 1 LEADER]: Becoming Leader. State: Replica: 32c82d7aafb447b0b94672baf0141973, State: Running, Role: LEADER
I20260926 01:08:46.887207 20660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:46.887285 20663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c82d7aafb447b0b94672baf0141973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } }
I20260926 01:08:46.887833 20665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c82d7aafb447b0b94672baf0141973. Latest consensus state: current_term: 1 leader_uuid: "32c82d7aafb447b0b94672baf0141973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c82d7aafb447b0b94672baf0141973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } } }
I20260926 01:08:46.887877 20664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c82d7aafb447b0b94672baf0141973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c82d7aafb447b0b94672baf0141973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } } }
I20260926 01:08:46.887935 20665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:46.887945 20664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:46.888401 20672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:46.888607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 20592
I20260926 01:08:46.888706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:46.888893 392 external_mini_cluster.cc:1428] Setting key bb8dba46260c52078ffd8d1d51bc2709
I20260926 01:08:46.889559 20672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:46.891320 20672 catalog_manager.cc:1347] Generated new cluster ID: f33f5c8a7ee0459790c51adf6442dd58
I20260926 01:08:46.891384 20672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:08:46 dist-test-slave-l73v krb5kdc[20579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:08:46.906354 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:46.889415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58517 (local address 127.0.0.1:46111)
0926 01:08:46.889578 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:46.889589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:46.889621 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:08:46.889755 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:46.889761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:46.889808 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:46.889942 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:46.890686 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:46.895151 (+ 4465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:46.895986 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:46.896460 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:46.896571 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:46.903139 (+ 6568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:46.903145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:46.903145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:46.903159 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:46.904759 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:46.905163 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:46.905167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:46.905167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:46.905260 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:46.905469 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:46.905470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:46.905470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:46.905744 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:46.905880 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:46.906012 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 01:08:46.906115 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260926 01:08:46.907442 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:46.912117 20672 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:46.912673 20672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:46.915747 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:46.908740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36851 (local address 127.0.0.1:46111)
0926 01:08:46.908792 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:46.908794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:46.908908 (+ 114us) server_negotiation.cc:407] Connection header received
0926 01:08:46.908963 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:46.908964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:46.908975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:46.909020 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:46.909697 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:46.910247 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:46.911020 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:46.911286 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:46.911344 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:46.912961 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:46.912962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:46.912963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:46.912970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:46.914557 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:46.914996 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:46.914997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:46.914997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:46.915051 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:46.915249 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:46.915249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:46.915250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:46.915369 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:46.915472 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:46.915592 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 01:08:46.915653 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:08:46.923694 20672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32c82d7aafb447b0b94672baf0141973: Generated new TSK 0
I20260926 01:08:46.923911 20672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:46.927470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 20592
W20260926 01:08:46.931496 20577 connection.cc:535] client connection to 127.0.0.1:46111 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46111): Connection reset by peer (error 104)
2026-09-26T01:08:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:46 dist-test-slave-l73v krb5kdc[20691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:46 dist-test-slave-l73v krb5kdc[20691](info): set up 2 sockets
Sep 26 01:08:46 dist-test-slave-l73v krb5kdc[20691](info): commencing operation
krb5kdc: starting...
W20260926 01:08:47.874236 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:47 dist-test-slave-l73v krb5kdc[20691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:47.998267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:48.080439 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:48.080682 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:48.080717 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:48.081990 20704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:48.082041 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:48.082060 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:48.082077 20704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:48.082091 20704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:48.083859 20704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36317
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:48.084133 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:48.084360 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:48.089109 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:48.089068 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:48.089149 20704 server_base.cc:1034] running on GCE node
W20260926 01:08:48.089069 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:48.089529 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:48.089828 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:48.090981 20704 hybrid_clock.cc:648] HybridClock initialized: now 1790384928090964 us; error 27 us; skew 500 ppm
Sep 26 01:08:48 dist-test-slave-l73v krb5kdc[20691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:48.093467 20704 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:48.093896 20704 webserver.cc:460] Webserver started at http://127.0.0.1:33281/ using document root <none> and password file <none>
I20260926 01:08:48.094141 20704 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:48.094192 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:48.094319 20704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:48.095417 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "4cdc387c149c48298628e5bd1a0b4793"
format_stamp: "Formatted at 2026-09-26 01:08:48 on dist-test-slave-l73v"
server_key: "5080a94b1a781338b40fe349df979401"
server_key_iv: "ce41fe6230888adfbf70c45ab022e819"
server_key_version: "encryptionkey@0"
I20260926 01:08:48.095750 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "4cdc387c149c48298628e5bd1a0b4793"
format_stamp: "Formatted at 2026-09-26 01:08:48 on dist-test-slave-l73v"
server_key: "5080a94b1a781338b40fe349df979401"
server_key_iv: "ce41fe6230888adfbf70c45ab022e819"
server_key_version: "encryptionkey@0"
I20260926 01:08:48.097116 20704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:08:48.097915 20719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:48.098201 20704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:48.098295 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "4cdc387c149c48298628e5bd1a0b4793"
format_stamp: "Formatted at 2026-09-26 01:08:48 on dist-test-slave-l73v"
server_key: "5080a94b1a781338b40fe349df979401"
server_key_iv: "ce41fe6230888adfbf70c45ab022e819"
server_key_version: "encryptionkey@0"
I20260926 01:08:48.098359 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:48.108929 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:48.111225 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:48.111373 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:48.115231 20704 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36317
I20260926 01:08:48.115315 20771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36317 every 8 connection(s)
I20260926 01:08:48.115609 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:48.116468 20772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:48.118861 20772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793: Bootstrap starting.
I20260926 01:08:48.119441 20772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:48.119694 20772 log.cc:824] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:48.120486 20772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793: No bootstrap required, opened a new log
I20260926 01:08:48.122274 20772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc387c149c48298628e5bd1a0b4793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36317 } }
I20260926 01:08:48.122494 20772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:48.122541 20772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdc387c149c48298628e5bd1a0b4793, State: Initialized, Role: FOLLOWER
I20260926 01:08:48.122705 20772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc387c149c48298628e5bd1a0b4793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36317 } }
I20260926 01:08:48.122792 20772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:48.122834 20772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:48.122886 20772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:48.123347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 20704
I20260926 01:08:48.123454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:48.123567 20772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc387c149c48298628e5bd1a0b4793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36317 } }
I20260926 01:08:48.123631 392 external_mini_cluster.cc:1428] Setting key 7aaa8361305239129e25c963f5bdbe2b
I20260926 01:08:48.123677 20772 leader_election.cc:302] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cdc387c149c48298628e5bd1a0b4793; no voters:
I20260926 01:08:48.123893 20772 leader_election.cc:288] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:48.123939 20775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:48.124063 20775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [term 1 LEADER]: Becoming Leader. State: Replica: 4cdc387c149c48298628e5bd1a0b4793, State: Running, Role: LEADER
I20260926 01:08:48.124250 20775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc387c149c48298628e5bd1a0b4793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36317 } }
I20260926 01:08:48.124327 20772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:48.124675 20777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cdc387c149c48298628e5bd1a0b4793. Latest consensus state: current_term: 1 leader_uuid: "4cdc387c149c48298628e5bd1a0b4793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc387c149c48298628e5bd1a0b4793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36317 } } }
I20260926 01:08:48.124783 20777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:48.124980 20776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cdc387c149c48298628e5bd1a0b4793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc387c149c48298628e5bd1a0b4793" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36317 } } }
I20260926 01:08:48.125046 20776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:48.125379 20781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:48.125900 20781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:48.127542 20781 catalog_manager.cc:1347] Generated new cluster ID: 62bd1da20706482d98f563f1fff996d7
I20260926 01:08:48.127601 20781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:48.129719 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:48.124091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59555 (local address 127.0.0.1:36317)
0926 01:08:48.124399 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:48.124410 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:48.124877 (+ 467us) server_negotiation.cc:407] Connection header received
0926 01:08:48.125013 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:48.125021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:48.125174 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:48.125362 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:48.125637 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:48.127012 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:48.127908 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:48.128340 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:48.128434 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:48.129107 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:48.129110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:48.129111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:48.129292 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:48.129321 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:48.129334 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:48.129406 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:48.129496 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 01:08:48.129526 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":170,"threads_started":1}
I20260926 01:08:48.135619 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:48.141808 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:48.138800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44215 (local address 127.0.0.1:36317)
0926 01:08:48.138900 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:48.138902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:48.139017 (+ 115us) server_negotiation.cc:407] Connection header received
0926 01:08:48.139080 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:48.139081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:48.139093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:48.139142 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:48.139278 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:48.139825 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:48.140579 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:48.140809 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:48.140867 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:48.141542 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:48.141544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:48.141544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:48.141550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:48.141568 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:48.141573 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:48.141616 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:48.141686 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 01:08:48.141701 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 01:08:48.145428 20781 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:48.146155 20781 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:48.154534 20781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cdc387c149c48298628e5bd1a0b4793: Generated new TSK 0
I20260926 01:08:48.154769 20781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:48.156849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 20704
W20260926 01:08:48.160642 20689 connection.cc:535] client connection to 127.0.0.1:36317 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36317): Connection reset by peer (error 104)
2026-09-26T01:08:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:48 dist-test-slave-l73v krb5kdc[20803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:48 dist-test-slave-l73v krb5kdc[20803](info): set up 2 sockets
Sep 26 01:08:48 dist-test-slave-l73v krb5kdc[20803](info): commencing operation
krb5kdc: starting...
W20260926 01:08:49.106230 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:49 dist-test-slave-l73v krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:49.225486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:49.308640 20816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:49.308887 20816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:49.308914 20816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:49.310344 20816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:49.310443 20816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:49.310458 20816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:49.310472 20816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:49.310482 20816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:49.312175 20816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:49.312409 20816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:49.312611 20816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:49.317652 20824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:49.317621 20822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:49.317678 20816 server_base.cc:1034] running on GCE node
W20260926 01:08:49.317616 20821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:49.318092 20816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:49.318423 20816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:49.319592 20816 hybrid_clock.cc:648] HybridClock initialized: now 1790384929319578 us; error 31 us; skew 500 ppm
Sep 26 01:08:49 dist-test-slave-l73v krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:49.322165 20816 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:49.322681 20816 webserver.cc:460] Webserver started at http://127.0.0.1:39563/ using document root <none> and password file <none>
I20260926 01:08:49.322904 20816 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:49.322953 20816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:49.323086 20816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:49.324168 20816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "b2fda62a355549909ab0790440df9183"
format_stamp: "Formatted at 2026-09-26 01:08:49 on dist-test-slave-l73v"
server_key: "2a326ff3aa12c62c85d265233a391852"
server_key_iv: "2cd37ad3e7b0a5980a9547409bc2ea81"
server_key_version: "encryptionkey@0"
I20260926 01:08:49.324509 20816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "b2fda62a355549909ab0790440df9183"
format_stamp: "Formatted at 2026-09-26 01:08:49 on dist-test-slave-l73v"
server_key: "2a326ff3aa12c62c85d265233a391852"
server_key_iv: "2cd37ad3e7b0a5980a9547409bc2ea81"
server_key_version: "encryptionkey@0"
I20260926 01:08:49.325901 20816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:08:49.326764 20831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:49.326977 20816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:08:49.327080 20816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "b2fda62a355549909ab0790440df9183"
format_stamp: "Formatted at 2026-09-26 01:08:49 on dist-test-slave-l73v"
server_key: "2a326ff3aa12c62c85d265233a391852"
server_key_iv: "2cd37ad3e7b0a5980a9547409bc2ea81"
server_key_version: "encryptionkey@0"
I20260926 01:08:49.327148 20816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:49.330590 20816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:49.332741 20816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:49.332876 20816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:49.336898 20816 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43315
I20260926 01:08:49.336934 20883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43315 every 8 connection(s)
I20260926 01:08:49.337285 20816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:49.338258 20884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:49.340250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 20816
I20260926 01:08:49.340368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:49.340569 392 external_mini_cluster.cc:1428] Setting key 001845d98038ec06aff84f0910133278
I20260926 01:08:49.340911 20884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183: Bootstrap starting.
I20260926 01:08:49.341607 20884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:49.341885 20884 log.cc:824] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:49.342708 20884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183: No bootstrap required, opened a new log
I20260926 01:08:49.343303 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:49.341132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60501 (local address 127.0.0.1:43315)
0926 01:08:49.341314 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:49.341323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:49.341413 (+ 90us) server_negotiation.cc:407] Connection header received
0926 01:08:49.341935 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:49.341942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:49.341979 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:49.342115 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:49.342790 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:49.342793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:49.342794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:49.342798 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:49.342854 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:49.342869 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:49.342957 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:49.343046 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 01:08:49.343087 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260926 01:08:49.344386 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:49.345038 20884 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fda62a355549909ab0790440df9183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43315 } }
I20260926 01:08:49.345198 20884 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:49.345224 20884 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2fda62a355549909ab0790440df9183, State: Initialized, Role: FOLLOWER
I20260926 01:08:49.345373 20884 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fda62a355549909ab0790440df9183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43315 } }
I20260926 01:08:49.345474 20884 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:49.345500 20884 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:49.345568 20884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:49.346196 20884 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fda62a355549909ab0790440df9183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43315 } }
I20260926 01:08:49.346299 20884 leader_election.cc:302] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2fda62a355549909ab0790440df9183; no voters:
I20260926 01:08:49.346513 20884 leader_election.cc:288] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:49.346541 20894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:49.346663 20894 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [term 1 LEADER]: Becoming Leader. State: Replica: b2fda62a355549909ab0790440df9183, State: Running, Role: LEADER
I20260926 01:08:49.346798 20894 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fda62a355549909ab0790440df9183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43315 } }
I20260926 01:08:49.346860 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:49.345724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52549 (local address 127.0.0.1:43315)
0926 01:08:49.345765 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:49.345767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:49.345871 (+ 104us) server_negotiation.cc:407] Connection header received
0926 01:08:49.345928 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:49.345928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:49.345933 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:49.345979 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:49.346546 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:49.346547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:49.346548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:49.346552 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:49.346568 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:49.346573 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:49.346620 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:49.346713 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 01:08:49.346726 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:08:49.346776 20884 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:49.347200 20894 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2fda62a355549909ab0790440df9183. Latest consensus state: current_term: 1 leader_uuid: "b2fda62a355549909ab0790440df9183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fda62a355549909ab0790440df9183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43315 } } }
I20260926 01:08:49.347273 20894 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:49.347388 20895 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2fda62a355549909ab0790440df9183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fda62a355549909ab0790440df9183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43315 } } }
I20260926 01:08:49.347455 20898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:49.347951 20898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:49.347460 20895 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:49.349408 20898 catalog_manager.cc:1347] Generated new cluster ID: 0ddddcd9477a4211bc42b467644a7c70
I20260926 01:08:49.349448 20898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:49.358929 20898 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:49.359119 20898 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:49.363336 20898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2fda62a355549909ab0790440df9183: Generated new TSK 0
I20260926 01:08:49.363529 20898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:49.368940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 20816
2026-09-26T01:08:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:49 dist-test-slave-l73v krb5kdc[20915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:49 dist-test-slave-l73v krb5kdc[20915](info): set up 2 sockets
Sep 26 01:08:49 dist-test-slave-l73v krb5kdc[20915](info): commencing operation
krb5kdc: starting...
W20260926 01:08:50.322288 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:50 dist-test-slave-l73v krb5kdc[20915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:50.450124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39609
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:50.532368 20928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:50.532605 20928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:50.532629 20928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:50.533838 20928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:50.533875 20928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:50.533888 20928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:50.533900 20928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:50.533910 20928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:50.535439 20928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:50.535673 20928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:50.535913 20928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:50.541035 20933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:50.541103 20936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:50.541244 20928 server_base.cc:1034] running on GCE node
W20260926 01:08:50.541106 20934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:50.541556 20928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:50.541884 20928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:50.543035 20928 hybrid_clock.cc:648] HybridClock initialized: now 1790384930543016 us; error 37 us; skew 500 ppm
Sep 26 01:08:50 dist-test-slave-l73v krb5kdc[20915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:50.545527 20928 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:50.545984 20928 webserver.cc:460] Webserver started at http://127.0.0.1:40683/ using document root <none> and password file <none>
I20260926 01:08:50.546185 20928 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:50.546228 20928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:50.546344 20928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:50.547464 20928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "42cfad4911bc448dbef3e2bab1d0761a"
format_stamp: "Formatted at 2026-09-26 01:08:50 on dist-test-slave-l73v"
server_key: "7cf323abcc5a63b3ad22dd86e7ca377c"
server_key_iv: "9e37189e8a71799147de9467c5b27183"
server_key_version: "encryptionkey@0"
I20260926 01:08:50.547794 20928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "42cfad4911bc448dbef3e2bab1d0761a"
format_stamp: "Formatted at 2026-09-26 01:08:50 on dist-test-slave-l73v"
server_key: "7cf323abcc5a63b3ad22dd86e7ca377c"
server_key_iv: "9e37189e8a71799147de9467c5b27183"
server_key_version: "encryptionkey@0"
I20260926 01:08:50.549180 20928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:08:50.550036 20943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:50.550194 20928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:50.550284 20928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "42cfad4911bc448dbef3e2bab1d0761a"
format_stamp: "Formatted at 2026-09-26 01:08:50 on dist-test-slave-l73v"
server_key: "7cf323abcc5a63b3ad22dd86e7ca377c"
server_key_iv: "9e37189e8a71799147de9467c5b27183"
server_key_version: "encryptionkey@0"
I20260926 01:08:50.550352 20928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:50.565604 20928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:50.567935 20928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:50.568094 20928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:50.572129 20928 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39609
I20260926 01:08:50.572165 20995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39609 every 8 connection(s)
I20260926 01:08:50.572517 20928 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:50.573429 20996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:50.575176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 20928
I20260926 01:08:50.575297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:50.575521 392 external_mini_cluster.cc:1428] Setting key 56d90981e67049998708f7accde01d56
I20260926 01:08:50.576156 20996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a: Bootstrap starting.
I20260926 01:08:50.576769 20996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:50.577042 20996 log.cc:824] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:50.577807 20996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a: No bootstrap required, opened a new log
I20260926 01:08:50.580032 20996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cfad4911bc448dbef3e2bab1d0761a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } }
I20260926 01:08:50.580274 20996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:50.580358 20996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cfad4911bc448dbef3e2bab1d0761a, State: Initialized, Role: FOLLOWER
I20260926 01:08:50.580619 20996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cfad4911bc448dbef3e2bab1d0761a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } }
I20260926 01:08:50.580749 20996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:50.580797 20996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:50.580856 20996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:50.581799 20996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cfad4911bc448dbef3e2bab1d0761a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } }
I20260926 01:08:50.581979 20996 leader_election.cc:302] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42cfad4911bc448dbef3e2bab1d0761a; no voters:
I20260926 01:08:50.582198 20996 leader_election.cc:288] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:50.582240 21001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:50.582415 21001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [term 1 LEADER]: Becoming Leader. State: Replica: 42cfad4911bc448dbef3e2bab1d0761a, State: Running, Role: LEADER
I20260926 01:08:50.582504 20996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:50.582516 21001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cfad4911bc448dbef3e2bab1d0761a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } }
I20260926 01:08:50.582899 21001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42cfad4911bc448dbef3e2bab1d0761a. Latest consensus state: current_term: 1 leader_uuid: "42cfad4911bc448dbef3e2bab1d0761a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cfad4911bc448dbef3e2bab1d0761a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } } }
I20260926 01:08:50.582988 21001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:50.582908 21003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42cfad4911bc448dbef3e2bab1d0761a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cfad4911bc448dbef3e2bab1d0761a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } } }
I20260926 01:08:50.583087 21003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a [sys.catalog]: This master's current role is: LEADER
Sep 26 01:08:50 dist-test-slave-l73v krb5kdc[20915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:08:50.583796 21007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:50.584254 21007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:50.586109 21007 catalog_manager.cc:1347] Generated new cluster ID: f3b53c7f799c4eeb8ce080f9a9de992e
I20260926 01:08:50.586158 21007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:50.588423 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:50.576105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57937 (local address 127.0.0.1:39609)
0926 01:08:50.576278 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:50.576288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:50.576317 (+ 29us) server_negotiation.cc:407] Connection header received
0926 01:08:50.576924 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:50.576933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:50.576991 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:50.577164 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:50.577843 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:50.578832 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:50.579929 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:50.580566 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:50.580670 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:50.584104 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:50.584110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:50.584110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:50.584125 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:50.586760 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:50.587196 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:50.587199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:50.587199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:50.587300 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:50.587493 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:50.587494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:50.587494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:50.587793 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:50.587929 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:50.588083 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 01:08:50.588217 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260926 01:08:50.589334 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:50.602109 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:50.590871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52711 (local address 127.0.0.1:39609)
0926 01:08:50.590908 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:50.590910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:50.594654 (+ 3744us) server_negotiation.cc:407] Connection header received
0926 01:08:50.594723 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:50.594724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:50.594737 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:50.594800 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:50.595504 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:50.596286 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:50.597322 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:50.597633 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:50.597688 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:50.599378 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:50.599379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:50.599380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:50.599388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:50.600902 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:50.601355 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:50.601356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:50.601356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:50.601406 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:50.601585 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:50.601586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:50.601586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:50.601710 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:50.601802 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:50.601924 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 01:08:50.602001 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:08:50.613749 21007 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:50.614472 21007 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:50.619189 21007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42cfad4911bc448dbef3e2bab1d0761a: Generated new TSK 0
I20260926 01:08:50.619452 21007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:50.626602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 20928
2026-09-26T01:08:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1258 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:50 dist-test-slave-l73v krb5kdc[21027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:50 dist-test-slave-l73v krb5kdc[21027](info): set up 2 sockets
Sep 26 01:08:50 dist-test-slave-l73v krb5kdc[21027](info): commencing operation
krb5kdc: starting...
W20260926 01:08:51.590240 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:51 dist-test-slave-l73v krb5kdc[21027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:51.709599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:51.790365 21040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:51.790671 21040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:51.790706 21040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:51.791985 21040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:51.792034 21040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:51.792053 21040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:51.792070 21040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:51.792085 21040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:51.793684 21040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:51.793970 21040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:51.794198 21040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:51.799221 21045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:51.799221 21046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:51.799314 21048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:51.799446 21040 server_base.cc:1034] running on GCE node
I20260926 01:08:51.799688 21040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:51.799980 21040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:51.801152 21040 hybrid_clock.cc:648] HybridClock initialized: now 1790384931801139 us; error 32 us; skew 500 ppm
Sep 26 01:08:51 dist-test-slave-l73v krb5kdc[21027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:51.803895 21040 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:51.804348 21040 webserver.cc:460] Webserver started at http://127.0.0.1:39685/ using document root <none> and password file <none>
I20260926 01:08:51.804580 21040 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:51.804631 21040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:51.804760 21040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:51.805833 21040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "bc69410e045b478ba06e248f902dc5c0"
format_stamp: "Formatted at 2026-09-26 01:08:51 on dist-test-slave-l73v"
server_key: "0eb06c31e4d56fe57e738e47425cd3e3"
server_key_iv: "c47dfb85bf22089644b23732b38723a5"
server_key_version: "encryptionkey@0"
I20260926 01:08:51.806190 21040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc69410e045b478ba06e248f902dc5c0"
format_stamp: "Formatted at 2026-09-26 01:08:51 on dist-test-slave-l73v"
server_key: "0eb06c31e4d56fe57e738e47425cd3e3"
server_key_iv: "c47dfb85bf22089644b23732b38723a5"
server_key_version: "encryptionkey@0"
I20260926 01:08:51.807534 21040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 01:08:51.808224 21055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:51.808379 21040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:08:51.808506 21040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "bc69410e045b478ba06e248f902dc5c0"
format_stamp: "Formatted at 2026-09-26 01:08:51 on dist-test-slave-l73v"
server_key: "0eb06c31e4d56fe57e738e47425cd3e3"
server_key_iv: "c47dfb85bf22089644b23732b38723a5"
server_key_version: "encryptionkey@0"
I20260926 01:08:51.808569 21040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:51.830411 21040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:51.832486 21040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:51.832626 21040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:51.836501 21040 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44477
I20260926 01:08:51.836545 21107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44477 every 8 connection(s)
I20260926 01:08:51.836876 21040 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:51.837733 21108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:51.840023 21108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0: Bootstrap starting.
I20260926 01:08:51.840572 21108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:51.840801 21108 log.cc:824] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:51.841584 21108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0: No bootstrap required, opened a new log
I20260926 01:08:51.843369 21108 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc69410e045b478ba06e248f902dc5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 01:08:51.843662 21108 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:51.843822 21108 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc69410e045b478ba06e248f902dc5c0, State: Initialized, Role: FOLLOWER
I20260926 01:08:51.843972 21108 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc69410e045b478ba06e248f902dc5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 01:08:51.844049 21108 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:51.844071 21108 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:51.844110 21108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:51.844497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 21040
I20260926 01:08:51.844604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:51.844780 392 external_mini_cluster.cc:1428] Setting key 249a461bceff45cf5459a46d6876f9c9
I20260926 01:08:51.844939 21108 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc69410e045b478ba06e248f902dc5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 01:08:51.845055 21108 leader_election.cc:302] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc69410e045b478ba06e248f902dc5c0; no voters:
I20260926 01:08:51.845206 21108 leader_election.cc:288] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:51.845270 21111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:51.845541 21111 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [term 1 LEADER]: Becoming Leader. State: Replica: bc69410e045b478ba06e248f902dc5c0, State: Running, Role: LEADER
I20260926 01:08:51.845606 21108 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:51.845636 21111 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc69410e045b478ba06e248f902dc5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 01:08:51.845979 21111 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc69410e045b478ba06e248f902dc5c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc69410e045b478ba06e248f902dc5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } } }
I20260926 01:08:51.846062 21111 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:51.846199 21111 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc69410e045b478ba06e248f902dc5c0. Latest consensus state: current_term: 1 leader_uuid: "bc69410e045b478ba06e248f902dc5c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc69410e045b478ba06e248f902dc5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } } }
I20260926 01:08:51.846249 21111 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:51.846529 21117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:51.847110 21117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:51.849442 21117 catalog_manager.cc:1347] Generated new cluster ID: b87fe5f280f449b4bff1a8d1cf48af47
I20260926 01:08:51.849511 21117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:51.851178 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:51.845419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57897 (local address 127.0.0.1:44477)
0926 01:08:51.845811 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:51.845820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:51.845849 (+ 29us) server_negotiation.cc:407] Connection header received
0926 01:08:51.845965 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:51.845973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:51.846035 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:51.846171 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:51.846559 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:51.847898 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:51.849021 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:51.849584 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:51.849700 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:51.850353 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:51.850357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:51.850357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:51.850616 (+ 259us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:51.850647 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:51.850660 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:51.850718 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:51.850831 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 01:08:51.851001 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":62,"threads_started":1}
I20260926 01:08:51.852103 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:51.861295 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:51.854805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55003 (local address 127.0.0.1:44477)
0926 01:08:51.854849 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:51.854851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:51.858663 (+ 3812us) server_negotiation.cc:407] Connection header received
0926 01:08:51.858723 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:51.858725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:51.858739 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:51.858789 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:51.858930 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:51.859437 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:51.860146 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:51.860346 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:51.860402 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:51.861049 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:51.861051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:51.861051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:51.861058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:51.861075 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:51.861079 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:51.861125 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:51.861174 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 01:08:51.861204 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:08:51.884387 21117 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:51.884946 21117 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:51.891603 21117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc69410e045b478ba06e248f902dc5c0: Generated new TSK 0
I20260926 01:08:51.891920 21117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:51.909685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 21040
2026-09-26T01:08:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:51 dist-test-slave-l73v krb5kdc[21139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:51 dist-test-slave-l73v krb5kdc[21139](info): set up 2 sockets
Sep 26 01:08:51 dist-test-slave-l73v krb5kdc[21139](info): commencing operation
krb5kdc: starting...
W20260926 01:08:52.874351 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:52 dist-test-slave-l73v krb5kdc[21139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:52.993701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:53.077404 21152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:53.077661 21152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:53.077698 21152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:53.079058 21152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:53.079118 21152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:53.079135 21152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:53.079154 21152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:53.079169 21152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:53.080984 21152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:53.081450 21152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:53.081729 21152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:53.087064 21158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:53.087065 21157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:53.087277 21152 server_base.cc:1034] running on GCE node
W20260926 01:08:53.087065 21160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:53.087584 21152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:53.087888 21152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:53.089023 21152 hybrid_clock.cc:648] HybridClock initialized: now 1790384933089010 us; error 31 us; skew 500 ppm
Sep 26 01:08:53 dist-test-slave-l73v krb5kdc[21139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:53.091523 21152 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:53.091986 21152 webserver.cc:460] Webserver started at http://127.0.0.1:39481/ using document root <none> and password file <none>
I20260926 01:08:53.092186 21152 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:53.092234 21152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:53.092355 21152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:53.093425 21152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "3d39fcca5169407db2c42cf38ce6f0c7"
format_stamp: "Formatted at 2026-09-26 01:08:53 on dist-test-slave-l73v"
server_key: "98361587699879de669b272fbb05f48f"
server_key_iv: "bb70378507ee8b32e18bcc36a7d8ce20"
server_key_version: "encryptionkey@0"
I20260926 01:08:53.093760 21152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d39fcca5169407db2c42cf38ce6f0c7"
format_stamp: "Formatted at 2026-09-26 01:08:53 on dist-test-slave-l73v"
server_key: "98361587699879de669b272fbb05f48f"
server_key_iv: "bb70378507ee8b32e18bcc36a7d8ce20"
server_key_version: "encryptionkey@0"
I20260926 01:08:53.095180 21152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:08:53.096019 21167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:53.096168 21152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:08:53.096257 21152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "3d39fcca5169407db2c42cf38ce6f0c7"
format_stamp: "Formatted at 2026-09-26 01:08:53 on dist-test-slave-l73v"
server_key: "98361587699879de669b272fbb05f48f"
server_key_iv: "bb70378507ee8b32e18bcc36a7d8ce20"
server_key_version: "encryptionkey@0"
I20260926 01:08:53.096326 21152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:53.099799 21152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:53.101931 21152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:53.102062 21152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:53.105839 21152 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38797
I20260926 01:08:53.105865 21219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38797 every 8 connection(s)
I20260926 01:08:53.106200 21152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:53.107271 21220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:53.108584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 21152
I20260926 01:08:53.108701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:53.108887 392 external_mini_cluster.cc:1428] Setting key b21c3fad43b253f44cb10d05912fdea5
I20260926 01:08:53.109894 21220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7: Bootstrap starting.
I20260926 01:08:53.110563 21220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:53.110836 21220 log.cc:824] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:53.111402 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:53.109434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40205 (local address 127.0.0.1:38797)
0926 01:08:53.109596 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:53.109604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:53.109630 (+ 26us) server_negotiation.cc:407] Connection header received
0926 01:08:53.110081 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:53.110089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:53.110127 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:53.110252 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:53.110971 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:53.110975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:53.110975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:53.110980 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:53.111018 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:53.111029 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:53.111102 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:53.111159 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 01:08:53.111200 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260926 01:08:53.111545 21220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7: No bootstrap required, opened a new log
I20260926 01:08:53.112406 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:53.113821 21220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d39fcca5169407db2c42cf38ce6f0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } }
I20260926 01:08:53.114048 21220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:53.114106 21220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d39fcca5169407db2c42cf38ce6f0c7, State: Initialized, Role: FOLLOWER
I20260926 01:08:53.114233 21220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d39fcca5169407db2c42cf38ce6f0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } }
I20260926 01:08:53.114336 21220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:53.114399 21220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:53.114455 21220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:53.115056 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:53.114007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58755 (local address 127.0.0.1:38797)
0926 01:08:53.114048 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:53.114050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:53.114106 (+ 56us) server_negotiation.cc:407] Connection header received
0926 01:08:53.114188 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:53.114189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:53.114194 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:53.114262 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:53.114855 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:53.114856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:53.114857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:53.114860 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:53.114875 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:53.114879 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:53.114922 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:53.114989 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 01:08:53.115000 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:08:53.115226 21220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d39fcca5169407db2c42cf38ce6f0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } }
I20260926 01:08:53.115338 21220 leader_election.cc:302] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d39fcca5169407db2c42cf38ce6f0c7; no voters:
I20260926 01:08:53.115506 21220 leader_election.cc:288] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:53.115545 21230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:53.115674 21230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [term 1 LEADER]: Becoming Leader. State: Replica: 3d39fcca5169407db2c42cf38ce6f0c7, State: Running, Role: LEADER
I20260926 01:08:53.115770 21220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:53.115792 21230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d39fcca5169407db2c42cf38ce6f0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } }
I20260926 01:08:53.116149 21231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d39fcca5169407db2c42cf38ce6f0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d39fcca5169407db2c42cf38ce6f0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } } }
I20260926 01:08:53.116158 21232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d39fcca5169407db2c42cf38ce6f0c7. Latest consensus state: current_term: 1 leader_uuid: "3d39fcca5169407db2c42cf38ce6f0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d39fcca5169407db2c42cf38ce6f0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } } }
I20260926 01:08:53.116338 21232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:53.116537 21231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:53.116540 21239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:53.117141 21239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:53.118667 21239 catalog_manager.cc:1347] Generated new cluster ID: 60c91fbdba2d4d67aabaebaeec75c8a9
I20260926 01:08:53.118722 21239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:53.131019 21239 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:53.131201 21239 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:53.137894 21239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d39fcca5169407db2c42cf38ce6f0c7: Generated new TSK 0
I20260926 01:08:53.138114 21239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:53.143672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 21152
2026-09-26T01:08:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:53 dist-test-slave-l73v krb5kdc[21251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:53 dist-test-slave-l73v krb5kdc[21251](info): set up 2 sockets
Sep 26 01:08:53 dist-test-slave-l73v krb5kdc[21251](info): commencing operation
krb5kdc: starting...
W20260926 01:08:54.098184 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:54 dist-test-slave-l73v krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:54.217505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:54.298053 21264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:54.298282 21264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:54.298306 21264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:54.299510 21264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:54.299558 21264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:54.299571 21264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:54.299583 21264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:54.299594 21264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:54.301120 21264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:54.301337 21264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:54.301527 21264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:54.306437 21270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:54.306423 21269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:54.306422 21272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:54.306464 21264 server_base.cc:1034] running on GCE node
I20260926 01:08:54.306928 21264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:54.307212 21264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:54.308386 21264 hybrid_clock.cc:648] HybridClock initialized: now 1790384934308365 us; error 38 us; skew 500 ppm
Sep 26 01:08:54 dist-test-slave-l73v krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:54.311174 21264 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:54.311663 21264 webserver.cc:460] Webserver started at http://127.0.0.1:33731/ using document root <none> and password file <none>
I20260926 01:08:54.311887 21264 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:54.311935 21264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:54.312062 21264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:54.313153 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "a30bac9623a04d28bf4d80ce03fb7f12"
format_stamp: "Formatted at 2026-09-26 01:08:54 on dist-test-slave-l73v"
server_key: "a9f46020303616f0c35a911acf85d6f6"
server_key_iv: "15f7331a223fec0386274e1f956ca1e1"
server_key_version: "encryptionkey@0"
I20260926 01:08:54.313477 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "a30bac9623a04d28bf4d80ce03fb7f12"
format_stamp: "Formatted at 2026-09-26 01:08:54 on dist-test-slave-l73v"
server_key: "a9f46020303616f0c35a911acf85d6f6"
server_key_iv: "15f7331a223fec0386274e1f956ca1e1"
server_key_version: "encryptionkey@0"
I20260926 01:08:54.314849 21264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 01:08:54.315730 21279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:54.315888 21264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:54.315974 21264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "a30bac9623a04d28bf4d80ce03fb7f12"
format_stamp: "Formatted at 2026-09-26 01:08:54 on dist-test-slave-l73v"
server_key: "a9f46020303616f0c35a911acf85d6f6"
server_key_iv: "15f7331a223fec0386274e1f956ca1e1"
server_key_version: "encryptionkey@0"
I20260926 01:08:54.316036 21264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:54.334028 21264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:54.336289 21264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:54.336438 21264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:54.340212 21264 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46423
I20260926 01:08:54.340246 21331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46423 every 8 connection(s)
I20260926 01:08:54.340570 21264 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:54.341498 21332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:54.342479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 21264
I20260926 01:08:54.342604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:54.342782 392 external_mini_cluster.cc:1428] Setting key 83de4a0a1a1c3cdae970bb30e5affcdc
I20260926 01:08:54.344229 21332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12: Bootstrap starting.
I20260926 01:08:54.344909 21332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:54.345219 21332 log.cc:824] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:54.345925 21332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12: No bootstrap required, opened a new log
I20260926 01:08:54.348253 21332 raft_consensus.cc:348] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30bac9623a04d28bf4d80ce03fb7f12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46423 } }
I20260926 01:08:54.348404 21332 raft_consensus.cc:374] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:54.348436 21332 raft_consensus.cc:729] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30bac9623a04d28bf4d80ce03fb7f12, State: Initialized, Role: FOLLOWER
I20260926 01:08:54.348578 21332 consensus_queue.cc:260] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30bac9623a04d28bf4d80ce03fb7f12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46423 } }
I20260926 01:08:54.348646 21332 raft_consensus.cc:388] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:54.348676 21332 raft_consensus.cc:482] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:54.348723 21332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:54.349305 21332 raft_consensus.cc:504] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30bac9623a04d28bf4d80ce03fb7f12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46423 } }
I20260926 01:08:54.349402 21332 leader_election.cc:302] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a30bac9623a04d28bf4d80ce03fb7f12; no voters:
I20260926 01:08:54.349570 21332 leader_election.cc:288] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:54.349608 21337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:54.349717 21337 raft_consensus.cc:686] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [term 1 LEADER]: Becoming Leader. State: Replica: a30bac9623a04d28bf4d80ce03fb7f12, State: Running, Role: LEADER
I20260926 01:08:54.349846 21337 consensus_queue.cc:237] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30bac9623a04d28bf4d80ce03fb7f12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46423 } }
I20260926 01:08:54.349845 21332 sys_catalog.cc:564] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 01:08:54 dist-test-slave-l73v krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:08:54.351017 21339 sys_catalog.cc:455] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a30bac9623a04d28bf4d80ce03fb7f12. Latest consensus state: current_term: 1 leader_uuid: "a30bac9623a04d28bf4d80ce03fb7f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30bac9623a04d28bf4d80ce03fb7f12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46423 } } }
I20260926 01:08:54.351109 21339 sys_catalog.cc:458] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:54.351488 21351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:54.351455 21338 sys_catalog.cc:455] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a30bac9623a04d28bf4d80ce03fb7f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30bac9623a04d28bf4d80ce03fb7f12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46423 } } }
I20260926 01:08:54.351630 21338 sys_catalog.cc:458] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:54.352018 21351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:54.353587 21351 catalog_manager.cc:1347] Generated new cluster ID: aebb915b40684fb3804e950c085dcff2
I20260926 01:08:54.353638 21351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:54.354326 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:54.343326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:52963 (local address 127.0.0.1:46423)
0926 01:08:54.343510 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:54.343519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:54.343547 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:08:54.344066 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:54.344072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:54.344125 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:54.344254 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:54.345035 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:54.346276 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:54.347433 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:54.347869 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:54.347964 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:54.350757 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:54.350762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:54.350762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:54.350776 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:54.352610 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:54.353108 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:54.353112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:54.353112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:54.353240 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:54.353459 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:54.353460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:54.353460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:54.353747 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:54.353898 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:54.354027 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 01:08:54.354126 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260926 01:08:54.355177 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:54.364437 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:54.358456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:39147 (local address 127.0.0.1:46423)
0926 01:08:54.358508 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:54.358510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:54.358523 (+ 13us) server_negotiation.cc:407] Connection header received
0926 01:08:54.358548 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:54.358549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:54.358561 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:54.358632 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:54.359332 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:54.359844 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:54.360596 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:54.360844 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:54.360901 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:54.362088 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:54.362089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:54.362090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:54.362097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:54.363364 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:54.363715 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:54.363716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:54.363716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:54.363766 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:54.363947 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:54.363948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:54.363948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:54.364052 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:54.364150 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:54.364285 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 01:08:54.364345 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 01:08:54.371361 21351 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:54.371973 21351 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:54.378057 21351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a30bac9623a04d28bf4d80ce03fb7f12: Generated new TSK 0
I20260926 01:08:54.378274 21351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:54.382660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 21264
W20260926 01:08:54.386571 21249 connection.cc:535] client connection to 127.0.0.1:46423 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46423): Connection reset by peer (error 104)
2026-09-26T01:08:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:54 dist-test-slave-l73v krb5kdc[21363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:54 dist-test-slave-l73v krb5kdc[21363](info): set up 2 sockets
Sep 26 01:08:54 dist-test-slave-l73v krb5kdc[21363](info): commencing operation
krb5kdc: starting...
W20260926 01:08:55.342278 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:55 dist-test-slave-l73v krb5kdc[21363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:55.466006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:55.546064 21376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:55.546293 21376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:55.546315 21376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:55.547565 21376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:55.547619 21376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:55.547632 21376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:55.547644 21376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:55.547654 21376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:55.549229 21376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:55.549453 21376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:55.549647 21376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:55.554418 21382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:55.554476 21381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:55.554515 21376 server_base.cc:1034] running on GCE node
W20260926 01:08:55.554482 21384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:55.554880 21376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:55.555183 21376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:55.556344 21376 hybrid_clock.cc:648] HybridClock initialized: now 1790384935556332 us; error 27 us; skew 500 ppm
Sep 26 01:08:55 dist-test-slave-l73v krb5kdc[21363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:55.558810 21376 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:55.559274 21376 webserver.cc:460] Webserver started at http://127.0.0.1:40361/ using document root <none> and password file <none>
I20260926 01:08:55.559520 21376 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:55.559573 21376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:55.559695 21376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:55.560753 21376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "2884ff2554d2489e967f87c1b5a2cebb"
format_stamp: "Formatted at 2026-09-26 01:08:55 on dist-test-slave-l73v"
server_key: "9f5d359ade1af44bbab0052eb62dbb47"
server_key_iv: "03fdd7b9a0713f12a29e33b9d67b9c28"
server_key_version: "encryptionkey@0"
I20260926 01:08:55.561080 21376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "2884ff2554d2489e967f87c1b5a2cebb"
format_stamp: "Formatted at 2026-09-26 01:08:55 on dist-test-slave-l73v"
server_key: "9f5d359ade1af44bbab0052eb62dbb47"
server_key_iv: "03fdd7b9a0713f12a29e33b9d67b9c28"
server_key_version: "encryptionkey@0"
I20260926 01:08:55.562314 21376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:55.563038 21391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:55.563194 21376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:08:55.563275 21376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "2884ff2554d2489e967f87c1b5a2cebb"
format_stamp: "Formatted at 2026-09-26 01:08:55 on dist-test-slave-l73v"
server_key: "9f5d359ade1af44bbab0052eb62dbb47"
server_key_iv: "03fdd7b9a0713f12a29e33b9d67b9c28"
server_key_version: "encryptionkey@0"
I20260926 01:08:55.563335 21376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:55.590365 21376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:55.592617 21376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:55.592763 21376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:55.596618 21376 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44073
I20260926 01:08:55.596982 21376 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:55.596968 21443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44073 every 8 connection(s)
I20260926 01:08:55.597641 21444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:55.600036 21444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb: Bootstrap starting.
I20260926 01:08:55.600752 21444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:55.601016 21444 log.cc:824] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:55.601256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 21376
I20260926 01:08:55.601383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:55.601562 392 external_mini_cluster.cc:1428] Setting key b5771fb0f430de61909a2f049c07916d
I20260926 01:08:55.601742 21444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb: No bootstrap required, opened a new log
I20260926 01:08:55.604292 21444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884ff2554d2489e967f87c1b5a2cebb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260926 01:08:55.604450 21444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:55.604493 21444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2884ff2554d2489e967f87c1b5a2cebb, State: Initialized, Role: FOLLOWER
I20260926 01:08:55.604628 21444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884ff2554d2489e967f87c1b5a2cebb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260926 01:08:55.604705 21444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:55.604738 21444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:55.604784 21444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:55.605506 21444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884ff2554d2489e967f87c1b5a2cebb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260926 01:08:55.605618 21444 leader_election.cc:302] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2884ff2554d2489e967f87c1b5a2cebb; no voters:
I20260926 01:08:55.605778 21444 leader_election.cc:288] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:55.605819 21449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:55.605966 21449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [term 1 LEADER]: Becoming Leader. State: Replica: 2884ff2554d2489e967f87c1b5a2cebb, State: Running, Role: LEADER
I20260926 01:08:55.606032 21444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:55.606073 21449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884ff2554d2489e967f87c1b5a2cebb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260926 01:08:55.606426 21449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2884ff2554d2489e967f87c1b5a2cebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884ff2554d2489e967f87c1b5a2cebb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } } }
I20260926 01:08:55.606568 21449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:55.606736 21449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2884ff2554d2489e967f87c1b5a2cebb. Latest consensus state: current_term: 1 leader_uuid: "2884ff2554d2489e967f87c1b5a2cebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884ff2554d2489e967f87c1b5a2cebb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } } }
I20260926 01:08:55.606837 21449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:55.607159 21457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:55.607412 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:55.602037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:50279 (local address 127.0.0.1:44073)
0926 01:08:55.602207 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:55.602216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:55.602246 (+ 30us) server_negotiation.cc:407] Connection header received
0926 01:08:55.603053 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:55.603061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:55.603116 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:55.603231 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:55.603459 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:55.604340 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:55.605380 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:55.605838 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:55.605943 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:55.606682 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:55.606686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:55.606686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:55.606882 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:55.606920 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:55.606936 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:55.607014 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:55.607203 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 01:08:55.607234 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260926 01:08:55.607671 21457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:55.608395 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:55.609227 21457 catalog_manager.cc:1347] Generated new cluster ID: cd391845ae044ee8bd1dc7aca10f5d78
I20260926 01:08:55.609274 21457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:55.613945 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:55.610476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:54821 (local address 127.0.0.1:44073)
0926 01:08:55.610516 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:55.610518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:55.610525 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:08:55.610549 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:55.610550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:55.610558 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:55.610610 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:55.610782 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:55.611482 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:55.612505 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:55.612816 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:55.612870 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:55.613600 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:55.613602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:55.613602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:55.613608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:55.613623 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:55.613626 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:55.613665 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:55.613850 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 01:08:55.613859 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:08:55.637190 21457 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:55.637704 21457 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:55.645917 21457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2884ff2554d2489e967f87c1b5a2cebb: Generated new TSK 0
I20260926 01:08:55.646224 21457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:55.653683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 21376
W20260926 01:08:55.657557 21361 connection.cc:535] client connection to 127.0.0.1:44073 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44073): Connection reset by peer (error 104)
2026-09-26T01:08:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1270 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:55 dist-test-slave-l73v krb5kdc[21475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:55 dist-test-slave-l73v krb5kdc[21475](info): set up 2 sockets
Sep 26 01:08:55 dist-test-slave-l73v krb5kdc[21475](info): commencing operation
krb5kdc: starting...
W20260926 01:08:56.602288 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:56 dist-test-slave-l73v krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:56.725278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:56.809403 21488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:56.809648 21488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:56.809675 21488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:56.811015 21488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:56.811074 21488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:56.811086 21488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:56.811097 21488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:56.811108 21488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:56.812739 21488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:56.812991 21488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:56.813228 21488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:56.818595 21496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:56.818652 21493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:56.818599 21494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:56.818912 21488 server_base.cc:1034] running on GCE node
I20260926 01:08:56.819160 21488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:56.819459 21488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:56.820632 21488 hybrid_clock.cc:648] HybridClock initialized: now 1790384936820617 us; error 45 us; skew 500 ppm
Sep 26 01:08:56 dist-test-slave-l73v krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:56.823372 21488 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:56.823949 21488 webserver.cc:460] Webserver started at http://127.0.0.1:38031/ using document root <none> and password file <none>
I20260926 01:08:56.824170 21488 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:56.824222 21488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:56.824361 21488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:56.825466 21488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "f1d904fa71d64b718fd921c2224ec3f2"
format_stamp: "Formatted at 2026-09-26 01:08:56 on dist-test-slave-l73v"
server_key: "825fab07384c97e4d182ded89840a026"
server_key_iv: "7c64ae8456fd776e1299578272684716"
server_key_version: "encryptionkey@0"
I20260926 01:08:56.825866 21488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1d904fa71d64b718fd921c2224ec3f2"
format_stamp: "Formatted at 2026-09-26 01:08:56 on dist-test-slave-l73v"
server_key: "825fab07384c97e4d182ded89840a026"
server_key_iv: "7c64ae8456fd776e1299578272684716"
server_key_version: "encryptionkey@0"
I20260926 01:08:56.827380 21488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:56.828208 21503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:56.828415 21488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:56.828513 21488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "f1d904fa71d64b718fd921c2224ec3f2"
format_stamp: "Formatted at 2026-09-26 01:08:56 on dist-test-slave-l73v"
server_key: "825fab07384c97e4d182ded89840a026"
server_key_iv: "7c64ae8456fd776e1299578272684716"
server_key_version: "encryptionkey@0"
I20260926 01:08:56.828583 21488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:56.832096 21488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:56.834144 21488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:56.834295 21488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:56.838205 21488 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34895
I20260926 01:08:56.838248 21555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34895 every 8 connection(s)
I20260926 01:08:56.838626 21488 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:56.839637 21556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:56.840027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 21488
I20260926 01:08:56.840131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:56.840303 392 external_mini_cluster.cc:1428] Setting key a875812d1266bdcefba8f4f2b26a8a0c
I20260926 01:08:56.843233 21556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2: Bootstrap starting.
I20260926 01:08:56.843856 21556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:56.844141 21556 log.cc:824] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:56.844974 21556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2: No bootstrap required, opened a new log
I20260926 01:08:56.847579 21556 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d904fa71d64b718fd921c2224ec3f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } }
I20260926 01:08:56.847808 21556 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:56.847870 21556 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d904fa71d64b718fd921c2224ec3f2, State: Initialized, Role: FOLLOWER
I20260926 01:08:56.848022 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:56.841344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:47069 (local address 127.0.0.1:34895)
0926 01:08:56.841530 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:56.841540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:56.841572 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:08:56.842071 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:56.842081 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:56.842117 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:56.842268 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:56.847126 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:56.847130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:56.847131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:56.847137 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:56.847192 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:56.847209 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:56.847300 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:56.847409 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 01:08:56.847569 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260926 01:08:56.848011 21556 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d904fa71d64b718fd921c2224ec3f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } }
I20260926 01:08:56.848095 21556 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:56.848126 21556 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:56.848174 21556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:56.849128 21556 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d904fa71d64b718fd921c2224ec3f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } }
I20260926 01:08:56.849252 21556 leader_election.cc:302] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1d904fa71d64b718fd921c2224ec3f2; no voters:
I20260926 01:08:56.849443 21556 leader_election.cc:288] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:56.849476 21561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:56.849472 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:56.849588 21561 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [term 1 LEADER]: Becoming Leader. State: Replica: f1d904fa71d64b718fd921c2224ec3f2, State: Running, Role: LEADER
I20260926 01:08:56.849673 21561 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d904fa71d64b718fd921c2224ec3f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } }
I20260926 01:08:56.849709 21556 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:56.850029 21561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1d904fa71d64b718fd921c2224ec3f2. Latest consensus state: current_term: 1 leader_uuid: "f1d904fa71d64b718fd921c2224ec3f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d904fa71d64b718fd921c2224ec3f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } } }
I20260926 01:08:56.850111 21561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:56.850036 21562 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1d904fa71d64b718fd921c2224ec3f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d904fa71d64b718fd921c2224ec3f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } } }
I20260926 01:08:56.850261 21562 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:56.851074 21565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:56.851626 21565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:56.853948 21565 catalog_manager.cc:1347] Generated new cluster ID: 77190030f30a4dbc981368e47aa3fcfd
I20260926 01:08:56.854004 21565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:56.854570 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:56.852631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:58855 (local address 127.0.0.1:34895)
0926 01:08:56.852670 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:56.852672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:56.853163 (+ 491us) server_negotiation.cc:407] Connection header received
0926 01:08:56.853217 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:56.853218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:56.853224 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:56.853287 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:56.854081 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:56.854082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:56.854083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:56.854088 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:56.854105 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:56.854111 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:56.854143 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:56.854482 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 01:08:56.854502 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:08:56.862164 21565 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:56.862454 21565 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:56.872083 21565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1d904fa71d64b718fd921c2224ec3f2: Generated new TSK 0
I20260926 01:08:56.872660 21565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:56.884541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 21488
2026-09-26T01:08:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:56 dist-test-slave-l73v krb5kdc[21587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:56 dist-test-slave-l73v krb5kdc[21587](info): set up 2 sockets
Sep 26 01:08:56 dist-test-slave-l73v krb5kdc[21587](info): commencing operation
krb5kdc: starting...
W20260926 01:08:57.846295 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:57 dist-test-slave-l73v krb5kdc[21587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:57.971509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:58.057344 21600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:58.057582 21600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:58.057608 21600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:58.058882 21600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:58.059032 21600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:58.059069 21600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:58.059082 21600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:58.059093 21600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:58.061208 21600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:58.061467 21600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:58.061652 21600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:58.066666 21605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:58.066669 21608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:58.066666 21606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:58.066684 21600 server_base.cc:1034] running on GCE node
I20260926 01:08:58.067140 21600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:58.067436 21600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:58.068593 21600 hybrid_clock.cc:648] HybridClock initialized: now 1790384938068573 us; error 37 us; skew 500 ppm
Sep 26 01:08:58 dist-test-slave-l73v krb5kdc[21587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:58.071130 21600 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:58.071576 21600 webserver.cc:460] Webserver started at http://127.0.0.1:37513/ using document root <none> and password file <none>
I20260926 01:08:58.071803 21600 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:58.071852 21600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:58.071990 21600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:58.073185 21600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "06a550b6c12744f7b6cadc02c9c399e1"
format_stamp: "Formatted at 2026-09-26 01:08:58 on dist-test-slave-l73v"
server_key: "a479fc20d0d15be42fec8c35fd114c41"
server_key_iv: "69e24e30b3d391d9bc7e1afa2d7cc089"
server_key_version: "encryptionkey@0"
I20260926 01:08:58.073562 21600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "06a550b6c12744f7b6cadc02c9c399e1"
format_stamp: "Formatted at 2026-09-26 01:08:58 on dist-test-slave-l73v"
server_key: "a479fc20d0d15be42fec8c35fd114c41"
server_key_iv: "69e24e30b3d391d9bc7e1afa2d7cc089"
server_key_version: "encryptionkey@0"
I20260926 01:08:58.075172 21600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 01:08:58.076195 21615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:58.076406 21600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:08:58.076493 21600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "06a550b6c12744f7b6cadc02c9c399e1"
format_stamp: "Formatted at 2026-09-26 01:08:58 on dist-test-slave-l73v"
server_key: "a479fc20d0d15be42fec8c35fd114c41"
server_key_iv: "69e24e30b3d391d9bc7e1afa2d7cc089"
server_key_version: "encryptionkey@0"
I20260926 01:08:58.076565 21600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:58.110034 21600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:58.112306 21600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:58.112460 21600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:58.116434 21600 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40335
I20260926 01:08:58.116503 21667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40335 every 8 connection(s)
I20260926 01:08:58.116832 21600 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:58.116986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 21600
I20260926 01:08:58.117089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:58.117267 392 external_mini_cluster.cc:1428] Setting key 8e53d60afafb71ce05c6a61fd73b666b
I20260926 01:08:58.117830 21668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:58.121114 21668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1: Bootstrap starting.
I20260926 01:08:58.121712 21668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:58.122020 21668 log.cc:824] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:58.122869 21668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1: No bootstrap required, opened a new log
Sep 26 01:08:58 dist-test-slave-l73v krb5kdc[21587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 01:08:58.125284 21668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a550b6c12744f7b6cadc02c9c399e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } }
I20260926 01:08:58.125488 21668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:58.125519 21668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a550b6c12744f7b6cadc02c9c399e1, State: Initialized, Role: FOLLOWER
I20260926 01:08:58.125638 21668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a550b6c12744f7b6cadc02c9c399e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } }
I20260926 01:08:58.125710 21668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:58.125733 21668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:58.125779 21668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:58.126513 21668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a550b6c12744f7b6cadc02c9c399e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } }
I20260926 01:08:58.126628 21668 leader_election.cc:302] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a550b6c12744f7b6cadc02c9c399e1; no voters:
I20260926 01:08:58.126806 21668 leader_election.cc:288] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:58.126984 21673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:58.127084 21668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:58.127146 21673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [term 1 LEADER]: Becoming Leader. State: Replica: 06a550b6c12744f7b6cadc02c9c399e1, State: Running, Role: LEADER
I20260926 01:08:58.127242 21673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a550b6c12744f7b6cadc02c9c399e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } }
I20260926 01:08:58.127884 21674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a550b6c12744f7b6cadc02c9c399e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a550b6c12744f7b6cadc02c9c399e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } } }
I20260926 01:08:58.127974 21674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:58.128119 21674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a550b6c12744f7b6cadc02c9c399e1. Latest consensus state: current_term: 1 leader_uuid: "06a550b6c12744f7b6cadc02c9c399e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a550b6c12744f7b6cadc02c9c399e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } } }
I20260926 01:08:58.128180 21674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1 [sys.catalog]: This master's current role is: LEADER
W20260926 01:08:58.128789 21688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:08:58.128854 21688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:08:58.128975 21684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:58.129164 21684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:58.130048 21671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:58.117877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:58391 (local address 127.0.0.1:40335)
0926 01:08:58.118054 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:58.118063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:58.118095 (+ 32us) server_negotiation.cc:407] Connection header received
0926 01:08:58.118658 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:58.118668 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:58.118739 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:58.118897 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:58.119535 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:58.120892 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:58.121754 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:58.122365 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:58.122518 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:58.125492 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:58.125496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:58.125497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:58.125512 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:58.127937 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:58.128581 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:58.128585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:58.128585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:58.128723 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:58.128985 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:58.128987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:58.128987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:58.129305 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:58.129465 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:58.129726 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 01:08:58.129839 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260926 01:08:58.130930 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:58.132642 21684 catalog_manager.cc:1347] Generated new cluster ID: 20312ba1af5f49659a6a7a7e5b2c3bad
I20260926 01:08:58.132694 21684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:58.139977 21671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:58.132150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:37485 (local address 127.0.0.1:40335)
0926 01:08:58.132183 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:58.132185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:58.132306 (+ 121us) server_negotiation.cc:407] Connection header received
0926 01:08:58.132364 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:58.132365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:58.132374 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:58.132426 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:58.133064 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:58.133848 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:58.134870 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:58.135197 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:58.135281 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:58.136743 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:58.136745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:58.136745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:08:58.136754 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:58.138445 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:58.138871 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:58.138872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:58.138873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:58.138953 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:08:58.139154 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:08:58.139154 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:08:58.139155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:08:58.139300 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:58.139429 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:58.139626 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 01:08:58.139698 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:08:58.145273 21684 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:58.145856 21684 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:58.151252 21684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06a550b6c12744f7b6cadc02c9c399e1: Generated new TSK 0
I20260926 01:08:58.151459 21684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:58.157636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 21600
W20260926 01:08:58.161721 21585 connection.cc:535] client connection to 127.0.0.1:40335 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40335): Connection reset by peer (error 104)
2026-09-26T01:08:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:58 dist-test-slave-l73v krb5kdc[21699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:58 dist-test-slave-l73v krb5kdc[21699](info): set up 2 sockets
Sep 26 01:08:58 dist-test-slave-l73v krb5kdc[21699](info): commencing operation
krb5kdc: starting...
W20260926 01:08:59.118299 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:08:59 dist-test-slave-l73v krb5kdc[21699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:08:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:08:59.237396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:08:59.318821 21712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:08:59.319051 21712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:08:59.319077 21712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:08:59.320325 21712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:08:59.320375 21712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:08:59.320389 21712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:08:59.320399 21712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:08:59.320410 21712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:08:59.322041 21712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:08:59.322295 21712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:08:59.322567 21712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:08:59.327401 21718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:59.327411 21720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:08:59.327430 21717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:08:59.327731 21712 server_base.cc:1034] running on GCE node
I20260926 01:08:59.327976 21712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:08:59.328261 21712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:08:59.329417 21712 hybrid_clock.cc:648] HybridClock initialized: now 1790384939329400 us; error 36 us; skew 500 ppm
Sep 26 01:08:59 dist-test-slave-l73v krb5kdc[21699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:08:59.331983 21712 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:08:59.332433 21712 webserver.cc:460] Webserver started at http://127.0.0.1:36849/ using document root <none> and password file <none>
I20260926 01:08:59.332635 21712 fs_manager.cc:359] Metadata directory not provided
I20260926 01:08:59.332686 21712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:08:59.332820 21712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:08:59.333884 21712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "ba0ee2906a9340bba1b34a0c3df1170e"
format_stamp: "Formatted at 2026-09-26 01:08:59 on dist-test-slave-l73v"
server_key: "879a6b7a9a83c712d675e6c5570363e0"
server_key_iv: "8e3468bccfc88857f5c8548265c4aa29"
server_key_version: "encryptionkey@0"
I20260926 01:08:59.334196 21712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba0ee2906a9340bba1b34a0c3df1170e"
format_stamp: "Formatted at 2026-09-26 01:08:59 on dist-test-slave-l73v"
server_key: "879a6b7a9a83c712d675e6c5570363e0"
server_key_iv: "8e3468bccfc88857f5c8548265c4aa29"
server_key_version: "encryptionkey@0"
I20260926 01:08:59.335469 21712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:08:59.336223 21727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:08:59.336410 21712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:08:59.336503 21712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "ba0ee2906a9340bba1b34a0c3df1170e"
format_stamp: "Formatted at 2026-09-26 01:08:59 on dist-test-slave-l73v"
server_key: "879a6b7a9a83c712d675e6c5570363e0"
server_key_iv: "8e3468bccfc88857f5c8548265c4aa29"
server_key_version: "encryptionkey@0"
I20260926 01:08:59.336565 21712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:08:59.344455 21712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:08:59.346778 21712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:08:59.346933 21712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:08:59.350642 21712 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46315
I20260926 01:08:59.350680 21779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46315 every 8 connection(s)
I20260926 01:08:59.351022 21712 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:08:59.352021 21780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:08:59.352105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 21712
I20260926 01:08:59.352210 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:08:59.352385 392 external_mini_cluster.cc:1428] Setting key adb04150b0a9ed38fc5fccef7d2949ca
I20260926 01:08:59.355536 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e: Bootstrap starting.
I20260926 01:08:59.356153 21780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e: Neither blocks nor log segments found. Creating new log.
I20260926 01:08:59.356411 21780 log.cc:824] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:08:59.357069 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e: No bootstrap required, opened a new log
I20260926 01:08:59.358263 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:59.352896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:49697 (local address 127.0.0.1:46315)
0926 01:08:59.353082 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:59.353092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:59.353126 (+ 34us) server_negotiation.cc:407] Connection header received
0926 01:08:59.353598 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:59.353608 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:59.353685 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:59.353841 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:59.354062 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:59.355433 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:59.356345 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:59.356790 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:59.356896 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:59.357597 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:59.357601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:59.357602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:59.357755 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:59.357785 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:59.357798 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:59.357866 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:59.358051 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 01:08:59.358075 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260926 01:08:59.359221 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:08:59.359253 21780 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0ee2906a9340bba1b34a0c3df1170e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260926 01:08:59.359417 21780 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:08:59.359458 21780 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0ee2906a9340bba1b34a0c3df1170e, State: Initialized, Role: FOLLOWER
I20260926 01:08:59.359575 21780 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0ee2906a9340bba1b34a0c3df1170e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260926 01:08:59.359653 21780 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:08:59.359690 21780 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:08:59.359736 21780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:08:59.360363 21780 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0ee2906a9340bba1b34a0c3df1170e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260926 01:08:59.360481 21780 leader_election.cc:302] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba0ee2906a9340bba1b34a0c3df1170e; no voters:
I20260926 01:08:59.360714 21780 leader_election.cc:288] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:08:59.360752 21790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:08:59.360940 21790 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [term 1 LEADER]: Becoming Leader. State: Replica: ba0ee2906a9340bba1b34a0c3df1170e, State: Running, Role: LEADER
I20260926 01:08:59.361001 21780 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:08:59.361057 21790 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0ee2906a9340bba1b34a0c3df1170e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260926 01:08:59.362084 21791 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba0ee2906a9340bba1b34a0c3df1170e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0ee2906a9340bba1b34a0c3df1170e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } } }
I20260926 01:08:59.362224 21791 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:59.362470 21805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:08:59.362877 21792 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba0ee2906a9340bba1b34a0c3df1170e. Latest consensus state: current_term: 1 leader_uuid: "ba0ee2906a9340bba1b34a0c3df1170e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0ee2906a9340bba1b34a0c3df1170e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } } }
I20260926 01:08:59.362972 21792 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e [sys.catalog]: This master's current role is: LEADER
I20260926 01:08:59.363138 21805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:08:59.364015 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:08:59.360524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:56645 (local address 127.0.0.1:46315)
0926 01:08:59.360560 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:08:59.360562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:08:59.360687 (+ 125us) server_negotiation.cc:407] Connection header received
0926 01:08:59.360747 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:08:59.360748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:08:59.360758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:08:59.360817 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:08:59.361046 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:59.361772 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:59.362804 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:08:59.363012 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:08:59.363073 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:08:59.363787 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:08:59.363788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:08:59.363788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:08:59.363793 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:08:59.363808 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:08:59.363812 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:08:59.363862 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:08:59.363936 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 01:08:59.363946 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:08:59.364626 21805 catalog_manager.cc:1347] Generated new cluster ID: 9d94a03138e6423b9765ddf57eeb4920
I20260926 01:08:59.364672 21805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:08:59.373481 21805 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:08:59.374190 21805 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:08:59.381983 21805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba0ee2906a9340bba1b34a0c3df1170e: Generated new TSK 0
I20260926 01:08:59.382213 21805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:08:59.386946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 21712
W20260926 01:08:59.391144 21697 connection.cc:535] client connection to 127.0.0.1:46315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46315): Connection reset by peer (error 104)
2026-09-26T01:08:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:08:59 dist-test-slave-l73v krb5kdc[21811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:08:59 dist-test-slave-l73v krb5kdc[21811](info): set up 2 sockets
Sep 26 01:08:59 dist-test-slave-l73v krb5kdc[21811](info): commencing operation
krb5kdc: starting...
W20260926 01:09:00.342317 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:00 dist-test-slave-l73v krb5kdc[21811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 01:09:00.466135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:00.551116 21824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:00.551337 21824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:00.551360 21824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:00.552508 21824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:00.552546 21824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:00.552558 21824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:00.552577 21824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:00.552587 21824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:00.554023 21824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:00.554222 21824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:00.554419 21824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:00.559258 21830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:00.559353 21832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:00.559366 21829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:00.559805 21824 server_base.cc:1034] running on GCE node
I20260926 01:09:00.560045 21824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:00.560341 21824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:00.561512 21824 hybrid_clock.cc:648] HybridClock initialized: now 1790384940561484 us; error 45 us; skew 500 ppm
Sep 26 01:09:00 dist-test-slave-l73v krb5kdc[21811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:00.564138 21824 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 01:09:00.564559 21824 webserver.cc:460] Webserver started at http://127.0.0.1:40073/ using document root <none> and password file <none>
I20260926 01:09:00.564780 21824 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:00.564831 21824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:00.564954 21824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:00.566009 21824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "718dd1345b9349cdb1337d8ef3e22bc8"
format_stamp: "Formatted at 2026-09-26 01:09:00 on dist-test-slave-l73v"
server_key: "4c0e74bfa7ecea424ba641c274fb254f"
server_key_iv: "02ced0b9d7ee0eea8239edf3d46392cf"
server_key_version: "encryptionkey@0"
I20260926 01:09:00.566337 21824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "718dd1345b9349cdb1337d8ef3e22bc8"
format_stamp: "Formatted at 2026-09-26 01:09:00 on dist-test-slave-l73v"
server_key: "4c0e74bfa7ecea424ba641c274fb254f"
server_key_iv: "02ced0b9d7ee0eea8239edf3d46392cf"
server_key_version: "encryptionkey@0"
I20260926 01:09:00.567607 21824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:00.568356 21839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:00.568534 21824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:00.568616 21824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "718dd1345b9349cdb1337d8ef3e22bc8"
format_stamp: "Formatted at 2026-09-26 01:09:00 on dist-test-slave-l73v"
server_key: "4c0e74bfa7ecea424ba641c274fb254f"
server_key_iv: "02ced0b9d7ee0eea8239edf3d46392cf"
server_key_version: "encryptionkey@0"
I20260926 01:09:00.568676 21824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:00.572243 21824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:00.574190 21824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:00.574328 21824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:00.578220 21824 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40371
I20260926 01:09:00.578289 21891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40371 every 8 connection(s)
I20260926 01:09:00.578627 21824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:00.579602 21892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:00.581035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 21824
I20260926 01:09:00.581193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:00.581378 392 external_mini_cluster.cc:1428] Setting key 66245e958dc6c068618c6be85ed10f65
I20260926 01:09:00.582172 21892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8: Bootstrap starting.
I20260926 01:09:00.582860 21892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:00.583173 21892 log.cc:824] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:00.584059 21892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8: No bootstrap required, opened a new log
I20260926 01:09:00.584542 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:00.581894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:47181 (local address 127.0.0.1:40371)
0926 01:09:00.582068 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:00.582083 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:00.582116 (+ 33us) server_negotiation.cc:407] Connection header received
0926 01:09:00.582909 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:00.582922 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:00.582983 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:00.583166 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:00.584120 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:00.584124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:00.584125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:09:00.584129 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:00.584177 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:09:00.584188 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:00.584258 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:00.584311 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 01:09:00.584350 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260926 01:09:00.585542 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:09:00.586267 21892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718dd1345b9349cdb1337d8ef3e22bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40371 } }
I20260926 01:09:00.586443 21892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:00.586489 21892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718dd1345b9349cdb1337d8ef3e22bc8, State: Initialized, Role: FOLLOWER
I20260926 01:09:00.586637 21892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718dd1345b9349cdb1337d8ef3e22bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40371 } }
I20260926 01:09:00.586728 21892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:09:00.586777 21892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:09:00.586825 21892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:00.587571 21892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718dd1345b9349cdb1337d8ef3e22bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40371 } }
I20260926 01:09:00.587685 21892 leader_election.cc:302] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 718dd1345b9349cdb1337d8ef3e22bc8; no voters:
I20260926 01:09:00.587867 21892 leader_election.cc:288] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:09:00.587893 21902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:00.587993 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:00.586920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:47411 (local address 127.0.0.1:40371)
0926 01:09:00.586957 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:00.586959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:00.587093 (+ 134us) server_negotiation.cc:407] Connection header received
0926 01:09:00.587164 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:00.587165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:00.587169 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:00.587220 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:00.587749 (+ 529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:00.587750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:00.587750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:09:00.587754 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:00.587770 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:09:00.587774 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:00.587826 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:00.587874 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 01:09:00.587887 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:09:00.588107 21902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [term 1 LEADER]: Becoming Leader. State: Replica: 718dd1345b9349cdb1337d8ef3e22bc8, State: Running, Role: LEADER
I20260926 01:09:00.588182 21892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:00.588253 21902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718dd1345b9349cdb1337d8ef3e22bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40371 } }
I20260926 01:09:00.588652 21902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 718dd1345b9349cdb1337d8ef3e22bc8. Latest consensus state: current_term: 1 leader_uuid: "718dd1345b9349cdb1337d8ef3e22bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718dd1345b9349cdb1337d8ef3e22bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40371 } } }
I20260926 01:09:00.588730 21902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:00.588819 21903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "718dd1345b9349cdb1337d8ef3e22bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718dd1345b9349cdb1337d8ef3e22bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40371 } } }
I20260926 01:09:00.589022 21911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:00.589515 21911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:00.589041 21903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:00.591105 21911 catalog_manager.cc:1347] Generated new cluster ID: cdb0fe528f494286a22c2459f201f151
I20260926 01:09:00.591157 21911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:09:00.604219 21911 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:00.604413 21911 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:00.615063 21911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 718dd1345b9349cdb1337d8ef3e22bc8: Generated new TSK 0
I20260926 01:09:00.615279 21911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:09:00.618845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 21824
2026-09-26T01:09:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:00 dist-test-slave-l73v krb5kdc[21923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:00 dist-test-slave-l73v krb5kdc[21923](info): set up 2 sockets
Sep 26 01:09:00 dist-test-slave-l73v krb5kdc[21923](info): commencing operation
krb5kdc: starting...
W20260926 01:09:01.586321 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:01 dist-test-slave-l73v krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:01.705678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:01.788359 21936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:01.788602 21936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:01.788638 21936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:01.789917 21936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:01.789971 21936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:01.789990 21936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:01.790005 21936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:01.790021 21936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:01.791647 21936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:01.791929 21936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:01.792147 21936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:09:01.797271 21936 server_base.cc:1034] running on GCE node
W20260926 01:09:01.797195 21944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:01.797202 21941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:01.797214 21942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:01.797753 21936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:01.798050 21936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:01.799196 21936 hybrid_clock.cc:648] HybridClock initialized: now 1790384941799186 us; error 32 us; skew 500 ppm
Sep 26 01:09:01 dist-test-slave-l73v krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:01.802157 21936 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:01.802695 21936 webserver.cc:460] Webserver started at http://127.0.98.62:35233/ using document root <none> and password file <none>
I20260926 01:09:01.802906 21936 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:01.802954 21936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:01.803088 21936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:01.804209 21936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "005ecd8a15e644e58d8f7efb0bbd0436"
format_stamp: "Formatted at 2026-09-26 01:09:01 on dist-test-slave-l73v"
server_key: "f589a4720ead41b50a028daa25ae1969"
server_key_iv: "bda71e59705b721749b0b2f8143f9117"
server_key_version: "encryptionkey@0"
I20260926 01:09:01.804555 21936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "005ecd8a15e644e58d8f7efb0bbd0436"
format_stamp: "Formatted at 2026-09-26 01:09:01 on dist-test-slave-l73v"
server_key: "f589a4720ead41b50a028daa25ae1969"
server_key_iv: "bda71e59705b721749b0b2f8143f9117"
server_key_version: "encryptionkey@0"
I20260926 01:09:01.806023 21936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:09:01.807016 21951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:01.807288 21936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:09:01.807400 21936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "005ecd8a15e644e58d8f7efb0bbd0436"
format_stamp: "Formatted at 2026-09-26 01:09:01 on dist-test-slave-l73v"
server_key: "f589a4720ead41b50a028daa25ae1969"
server_key_iv: "bda71e59705b721749b0b2f8143f9117"
server_key_version: "encryptionkey@0"
I20260926 01:09:01.807476 21936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:01.827297 21936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:01.829339 21936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:01.829483 21936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:01.833384 21936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36201
I20260926 01:09:01.833424 22003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36201 every 8 connection(s)
I20260926 01:09:01.833783 21936 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:01.834697 22004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:01.837276 22004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436: Bootstrap starting.
I20260926 01:09:01.838008 22004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:01.838302 22004 log.cc:824] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:01.839174 22004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436: No bootstrap required, opened a new log
I20260926 01:09:01.840896 22004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005ecd8a15e644e58d8f7efb0bbd0436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36201 } }
I20260926 01:09:01.841030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 21936
I20260926 01:09:01.841081 22004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:01.841106 22004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 005ecd8a15e644e58d8f7efb0bbd0436, State: Initialized, Role: FOLLOWER
I20260926 01:09:01.841130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:01.841248 22004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005ecd8a15e644e58d8f7efb0bbd0436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36201 } }
I20260926 01:09:01.841320 22004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:09:01.841323 392 external_mini_cluster.cc:1428] Setting key dfa38e5824876b9f2028a7800f843343
I20260926 01:09:01.841356 22004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:09:01.841406 22004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:01.842705 22004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005ecd8a15e644e58d8f7efb0bbd0436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36201 } }
I20260926 01:09:01.842828 22004 leader_election.cc:302] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 005ecd8a15e644e58d8f7efb0bbd0436; no voters:
I20260926 01:09:01.842996 22004 leader_election.cc:288] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:09:01.843035 22009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:01.843135 22009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [term 1 LEADER]: Becoming Leader. State: Replica: 005ecd8a15e644e58d8f7efb0bbd0436, State: Running, Role: LEADER
I20260926 01:09:01.843292 22004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:01.843279 22009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005ecd8a15e644e58d8f7efb0bbd0436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36201 } }
I20260926 01:09:01.843762 22010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "005ecd8a15e644e58d8f7efb0bbd0436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005ecd8a15e644e58d8f7efb0bbd0436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36201 } } }
I20260926 01:09:01.843865 22010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:01.844063 22011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 005ecd8a15e644e58d8f7efb0bbd0436. Latest consensus state: current_term: 1 leader_uuid: "005ecd8a15e644e58d8f7efb0bbd0436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005ecd8a15e644e58d8f7efb0bbd0436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36201 } } }
I20260926 01:09:01.844388 22011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:01.844537 22017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:01.845063 22017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:01.846989 22017 catalog_manager.cc:1347] Generated new cluster ID: 626031a52931420096520d26cc224978
I20260926 01:09:01.847040 22017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 01:09:01 dist-test-slave-l73v krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:01.854525 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:01.842029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51713 (local address 127.0.98.62:36201)
0926 01:09:01.842266 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:01.842275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:01.842303 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:09:01.842448 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:01.842456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:01.842520 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:01.842660 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:01.843458 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:01.845020 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:01.846071 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:01.846633 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:01.846758 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:01.849733 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:01.849738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:01.849739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:01.849755 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:01.852535 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:01.853098 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:01.853103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:01.853103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:01.853255 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:01.853486 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:01.853487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:01.853487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:01.853866 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:01.854049 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:01.854135 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 01:09:01.854246 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":136,"threads_started":1}
I20260926 01:09:01.855911 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:09:01.865923 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:01.857887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48787 (local address 127.0.98.62:36201)
0926 01:09:01.857937 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:01.857939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:01.858041 (+ 102us) server_negotiation.cc:407] Connection header received
0926 01:09:01.858114 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:01.858115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:01.858127 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:01.858306 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:01.859034 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:01.859914 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:01.861064 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:01.861411 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:01.861482 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:01.863012 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:01.863014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:01.863014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:01.863022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:01.864619 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:01.865105 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:01.865106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:01.865107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:01.865172 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:01.865384 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:01.865384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:01.865385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:01.865514 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:01.865619 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:01.865756 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 01:09:01.865823 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 01:09:01.871258 22017 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:01.871876 22017 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:01.879755 22017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 005ecd8a15e644e58d8f7efb0bbd0436: Generated new TSK 0
I20260926 01:09:01.880012 22017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:09:01.883945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 21936
W20260926 01:09:01.888206 21921 connection.cc:535] client connection to 127.0.98.62:36201 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36201): Connection reset by peer (error 104)
2026-09-26T01:09:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:01 dist-test-slave-l73v krb5kdc[22035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:01 dist-test-slave-l73v krb5kdc[22035](info): set up 2 sockets
Sep 26 01:09:01 dist-test-slave-l73v krb5kdc[22035](info): commencing operation
krb5kdc: starting...
W20260926 01:09:02.838361 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:02 dist-test-slave-l73v krb5kdc[22035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:02.958000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42305
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:03.040270 22048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:03.040515 22048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:03.040556 22048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:03.041831 22048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:03.041878 22048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:03.041898 22048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:03.041913 22048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:03.041930 22048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:03.043617 22048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:03.043881 22048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:03.044099 22048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:09:03.048995 22048 server_base.cc:1034] running on GCE node
W20260926 01:09:03.049031 22056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:03.048964 22054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:03.048969 22053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:03.049562 22048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:03.049894 22048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:03.051047 22048 hybrid_clock.cc:648] HybridClock initialized: now 1790384943051022 us; error 42 us; skew 500 ppm
Sep 26 01:09:03 dist-test-slave-l73v krb5kdc[22035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:03.059412 22048 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:03.059897 22048 webserver.cc:460] Webserver started at http://127.0.98.62:35843/ using document root <none> and password file <none>
I20260926 01:09:03.060104 22048 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:03.060153 22048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:03.060281 22048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:03.061350 22048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "f985bc443a66412096bcf616eebe2a3b"
format_stamp: "Formatted at 2026-09-26 01:09:03 on dist-test-slave-l73v"
server_key: "fc36634e568bc01386376e172604ed31"
server_key_iv: "56e9623b1f50fda05fd35c5d04348aa2"
server_key_version: "encryptionkey@0"
I20260926 01:09:03.061698 22048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "f985bc443a66412096bcf616eebe2a3b"
format_stamp: "Formatted at 2026-09-26 01:09:03 on dist-test-slave-l73v"
server_key: "fc36634e568bc01386376e172604ed31"
server_key_iv: "56e9623b1f50fda05fd35c5d04348aa2"
server_key_version: "encryptionkey@0"
I20260926 01:09:03.063127 22048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:09:03.063930 22063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:03.064105 22048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:03.064195 22048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "f985bc443a66412096bcf616eebe2a3b"
format_stamp: "Formatted at 2026-09-26 01:09:03 on dist-test-slave-l73v"
server_key: "fc36634e568bc01386376e172604ed31"
server_key_iv: "56e9623b1f50fda05fd35c5d04348aa2"
server_key_version: "encryptionkey@0"
I20260926 01:09:03.064260 22048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:03.080686 22048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:03.083047 22048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:03.083192 22048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:03.087011 22048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42305
I20260926 01:09:03.087051 22115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42305 every 8 connection(s)
I20260926 01:09:03.087407 22048 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:03.088292 22116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:03.090579 22116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b: Bootstrap starting.
I20260926 01:09:03.091123 22116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:03.091349 22116 log.cc:824] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:03.091969 22116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b: No bootstrap required, opened a new log
I20260926 01:09:03.093067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 22048
I20260926 01:09:03.093214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:03.093376 392 external_mini_cluster.cc:1428] Setting key d61c49647ca1ea39ac1d443d0c2ec71b
I20260926 01:09:03.093607 22116 raft_consensus.cc:348] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f985bc443a66412096bcf616eebe2a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42305 } }
I20260926 01:09:03.093873 22116 raft_consensus.cc:374] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:03.093933 22116 raft_consensus.cc:729] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f985bc443a66412096bcf616eebe2a3b, State: Initialized, Role: FOLLOWER
I20260926 01:09:03.094058 22116 consensus_queue.cc:260] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f985bc443a66412096bcf616eebe2a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42305 } }
I20260926 01:09:03.094139 22116 raft_consensus.cc:388] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:09:03.094169 22116 raft_consensus.cc:482] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:09:03.094210 22116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:03.095206 22116 raft_consensus.cc:504] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f985bc443a66412096bcf616eebe2a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42305 } }
I20260926 01:09:03.095302 22116 leader_election.cc:302] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f985bc443a66412096bcf616eebe2a3b; no voters:
I20260926 01:09:03.095456 22116 leader_election.cc:288] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:09:03.095489 22121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:03.095587 22121 raft_consensus.cc:686] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [term 1 LEADER]: Becoming Leader. State: Replica: f985bc443a66412096bcf616eebe2a3b, State: Running, Role: LEADER
I20260926 01:09:03.095727 22116 sys_catalog.cc:564] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:03.095718 22121 consensus_queue.cc:237] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f985bc443a66412096bcf616eebe2a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42305 } }
I20260926 01:09:03.096912 22122 sys_catalog.cc:455] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f985bc443a66412096bcf616eebe2a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f985bc443a66412096bcf616eebe2a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42305 } } }
I20260926 01:09:03.097021 22122 sys_catalog.cc:458] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:03.097404 22123 sys_catalog.cc:455] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f985bc443a66412096bcf616eebe2a3b. Latest consensus state: current_term: 1 leader_uuid: "f985bc443a66412096bcf616eebe2a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f985bc443a66412096bcf616eebe2a3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42305 } } }
W20260926 01:09:03.097756 22135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 01:09:03.097910 22123 sys_catalog.cc:458] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b [sys.catalog]: This master's current role is: LEADER
W20260926 01:09:03.097951 22135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:09:03.098208 22136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:03.098433 22136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:03.098930 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:03.093902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60701 (local address 127.0.98.62:42305)
0926 01:09:03.094073 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:03.094080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:03.094117 (+ 37us) server_negotiation.cc:407] Connection header received
0926 01:09:03.094226 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:03.094233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:03.094290 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:03.094445 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:03.094634 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:03.095580 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:03.096492 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:03.097137 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:03.097268 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:03.098205 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:03.098209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:03.098209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:09:03.098433 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:03.098475 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:09:03.098491 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:03.098576 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:03.098685 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 01:09:03.098715 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260926 01:09:03.099932 22136 catalog_manager.cc:1347] Generated new cluster ID: d940a85c03b44df89ad9371845db56a1
I20260926 01:09:03.099979 22136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:09:03.100035 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:09:03.105086 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:03.101426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54357 (local address 127.0.98.62:42305)
0926 01:09:03.101484 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:03.101486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:03.101701 (+ 215us) server_negotiation.cc:407] Connection header received
0926 01:09:03.101736 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:03.101737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:03.101748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:03.101816 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:03.101940 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:03.102734 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:03.103744 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:03.104059 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:03.104135 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:03.104828 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:03.104830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:03.104830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:09:03.104836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:03.104854 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:09:03.104859 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:03.104923 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:03.104983 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 01:09:03.104997 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 01:09:03.123478 22136 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:03.124078 22136 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:03.129130 22136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f985bc443a66412096bcf616eebe2a3b: Generated new TSK 0
I20260926 01:09:03.129364 22136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:09:03.138952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 22048
W20260926 01:09:03.142973 22033 connection.cc:535] client connection to 127.0.98.62:42305 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42305): Connection reset by peer (error 104)
2026-09-26T01:09:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:03 dist-test-slave-l73v krb5kdc[22147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:03 dist-test-slave-l73v krb5kdc[22147](info): set up 2 sockets
Sep 26 01:09:03 dist-test-slave-l73v krb5kdc[22147](info): commencing operation
krb5kdc: starting...
W20260926 01:09:04.094274 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:04 dist-test-slave-l73v krb5kdc[22147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:04.214033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37431
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:04.295053 22160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:04.295296 22160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:04.295333 22160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:04.296614 22160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:04.296669 22160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:04.296686 22160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:04.296703 22160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:04.296718 22160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:04.298367 22160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:04.298677 22160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:04.298895 22160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:04.303790 22165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:04.303793 22166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:04.303833 22168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:04.303823 22160 server_base.cc:1034] running on GCE node
I20260926 01:09:04.304234 22160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:04.304529 22160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:04.305707 22160 hybrid_clock.cc:648] HybridClock initialized: now 1790384944305694 us; error 33 us; skew 500 ppm
Sep 26 01:09:04 dist-test-slave-l73v krb5kdc[22147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:04.308363 22160 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:04.308825 22160 webserver.cc:460] Webserver started at http://127.0.98.62:41041/ using document root <none> and password file <none>
I20260926 01:09:04.309024 22160 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:04.309070 22160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:04.309197 22160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:04.310249 22160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "22930dda2b874f3682d74e08fdadcf55"
format_stamp: "Formatted at 2026-09-26 01:09:04 on dist-test-slave-l73v"
server_key: "2844290ca815405b7076b2b969d27173"
server_key_iv: "03b718dedb90c2a5fc3c29e035e907f7"
server_key_version: "encryptionkey@0"
I20260926 01:09:04.310616 22160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "22930dda2b874f3682d74e08fdadcf55"
format_stamp: "Formatted at 2026-09-26 01:09:04 on dist-test-slave-l73v"
server_key: "2844290ca815405b7076b2b969d27173"
server_key_iv: "03b718dedb90c2a5fc3c29e035e907f7"
server_key_version: "encryptionkey@0"
I20260926 01:09:04.311861 22160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:04.312568 22175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:04.312732 22160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:04.312814 22160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "22930dda2b874f3682d74e08fdadcf55"
format_stamp: "Formatted at 2026-09-26 01:09:04 on dist-test-slave-l73v"
server_key: "2844290ca815405b7076b2b969d27173"
server_key_iv: "03b718dedb90c2a5fc3c29e035e907f7"
server_key_version: "encryptionkey@0"
I20260926 01:09:04.312875 22160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:04.316099 22160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:04.317899 22160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:04.318022 22160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:04.321764 22160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37431
I20260926 01:09:04.321805 22227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37431 every 8 connection(s)
I20260926 01:09:04.322099 22160 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:04.323127 22228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:04.325532 22228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55: Bootstrap starting.
I20260926 01:09:04.326090 22228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:04.326339 22228 log.cc:824] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:04.327068 22228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55: No bootstrap required, opened a new log
I20260926 01:09:04.328765 22228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22930dda2b874f3682d74e08fdadcf55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260926 01:09:04.328974 22228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:04.328985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 22160
I20260926 01:09:04.329015 22228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22930dda2b874f3682d74e08fdadcf55, State: Initialized, Role: FOLLOWER
I20260926 01:09:04.329095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:04.329190 22228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22930dda2b874f3682d74e08fdadcf55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260926 01:09:04.329279 22228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:09:04.329294 392 external_mini_cluster.cc:1428] Setting key 026e0326823f6a715a5c989343f85b59
I20260926 01:09:04.329335 22228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:09:04.329389 22228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:04.330103 22228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22930dda2b874f3682d74e08fdadcf55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260926 01:09:04.330204 22228 leader_election.cc:302] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22930dda2b874f3682d74e08fdadcf55; no voters:
I20260926 01:09:04.330412 22228 leader_election.cc:288] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:09:04.330430 22233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:04.330652 22233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [term 1 LEADER]: Becoming Leader. State: Replica: 22930dda2b874f3682d74e08fdadcf55, State: Running, Role: LEADER
I20260926 01:09:04.330709 22228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:04.330776 22233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22930dda2b874f3682d74e08fdadcf55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260926 01:09:04.331146 22234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22930dda2b874f3682d74e08fdadcf55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22930dda2b874f3682d74e08fdadcf55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } }
I20260926 01:09:04.331138 22233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22930dda2b874f3682d74e08fdadcf55. Latest consensus state: current_term: 1 leader_uuid: "22930dda2b874f3682d74e08fdadcf55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22930dda2b874f3682d74e08fdadcf55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } }
I20260926 01:09:04.331328 22233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:04.331313 22234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:04.331733 22242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:04.332201 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:04.329779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38437 (local address 127.0.98.62:37431)
0926 01:09:04.330423 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:04.330433 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:04.330473 (+ 40us) server_negotiation.cc:407] Connection header received
0926 01:09:04.330604 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:04.330611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:04.330643 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:04.330827 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:04.331516 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:04.331520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:04.331521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:09:04.331526 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:04.331575 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:09:04.331593 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:04.331898 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:04.331948 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 01:09:04.331996 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":508,"threads_started":1}
I20260926 01:09:04.332660 22242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:04.333154 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:09:04.334371 22242 catalog_manager.cc:1347] Generated new cluster ID: cc869fa63128446aa16b6f6ff930aea8
I20260926 01:09:04.334450 22242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:09:04.337638 22242 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:04.337810 22242 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:04.338573 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:04.334738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32861 (local address 127.0.98.62:37431)
0926 01:09:04.334777 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:04.334779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:04.337491 (+ 2712us) server_negotiation.cc:407] Connection header received
0926 01:09:04.337521 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:04.337521 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:04.337533 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:04.337578 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:04.338304 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:04.338305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:04.338306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:09:04.338311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:04.338330 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:09:04.338336 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:04.338434 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:04.338481 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 01:09:04.338495 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:09:04.345032 22242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22930dda2b874f3682d74e08fdadcf55: Generated new TSK 0
I20260926 01:09:04.345219 22242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:09:04.347395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 22160
2026-09-26T01:09:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:04 dist-test-slave-l73v krb5kdc[22259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:04 dist-test-slave-l73v krb5kdc[22259](info): set up 2 sockets
Sep 26 01:09:04 dist-test-slave-l73v krb5kdc[22259](info): commencing operation
krb5kdc: starting...
W20260926 01:09:05.302273 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:05 dist-test-slave-l73v krb5kdc[22259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:05.425082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34773
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:05.505100 22272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:05.505344 22272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:05.505369 22272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:05.506635 22272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:05.506685 22272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:05.506699 22272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:05.506711 22272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:05.506721 22272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:05.508276 22272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:05.508525 22272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:05.508755 22272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:05.513628 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:05.513680 22277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:05.513612 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:05.513835 22272 server_base.cc:1034] running on GCE node
I20260926 01:09:05.514189 22272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:05.514537 22272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:05.515730 22272 hybrid_clock.cc:648] HybridClock initialized: now 1790384945515699 us; error 43 us; skew 500 ppm
Sep 26 01:09:05 dist-test-slave-l73v krb5kdc[22259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:05.518487 22272 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:05.518986 22272 webserver.cc:460] Webserver started at http://127.0.98.62:41275/ using document root <none> and password file <none>
I20260926 01:09:05.519238 22272 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:05.519284 22272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:05.519407 22272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:05.520473 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "43d44cccd80945079061c1e47fcdb161"
format_stamp: "Formatted at 2026-09-26 01:09:05 on dist-test-slave-l73v"
server_key: "abab6a6508f04fcce446c235e8689fbd"
server_key_iv: "fda0c41b42b702c33e92d6a71b3d55be"
server_key_version: "encryptionkey@0"
I20260926 01:09:05.520807 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "43d44cccd80945079061c1e47fcdb161"
format_stamp: "Formatted at 2026-09-26 01:09:05 on dist-test-slave-l73v"
server_key: "abab6a6508f04fcce446c235e8689fbd"
server_key_iv: "fda0c41b42b702c33e92d6a71b3d55be"
server_key_version: "encryptionkey@0"
I20260926 01:09:05.522135 22272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:09:05.523020 22287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:05.523190 22272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 01:09:05.523277 22272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "43d44cccd80945079061c1e47fcdb161"
format_stamp: "Formatted at 2026-09-26 01:09:05 on dist-test-slave-l73v"
server_key: "abab6a6508f04fcce446c235e8689fbd"
server_key_iv: "fda0c41b42b702c33e92d6a71b3d55be"
server_key_version: "encryptionkey@0"
I20260926 01:09:05.523345 22272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:05.539629 22272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:05.541829 22272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:05.541986 22272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:05.545982 22272 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34773
I20260926 01:09:05.546037 22339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34773 every 8 connection(s)
I20260926 01:09:05.546367 22272 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:05.547261 22340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:05.549436 22340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161: Bootstrap starting.
I20260926 01:09:05.549978 22340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:05.550135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 22272
I20260926 01:09:05.550237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:05.550263 22340 log.cc:824] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:05.550453 392 external_mini_cluster.cc:1428] Setting key 8181404f22da65e6ce6ce81fc242b597
I20260926 01:09:05.551098 22340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161: No bootstrap required, opened a new log
I20260926 01:09:05.553865 22340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d44cccd80945079061c1e47fcdb161" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } }
I20260926 01:09:05.554052 22340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:05.554091 22340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43d44cccd80945079061c1e47fcdb161, State: Initialized, Role: FOLLOWER
I20260926 01:09:05.554243 22340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d44cccd80945079061c1e47fcdb161" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } }
I20260926 01:09:05.554318 22340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:09:05.554351 22340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:09:05.554425 22340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:05.555075 22340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d44cccd80945079061c1e47fcdb161" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } }
I20260926 01:09:05.555199 22340 leader_election.cc:302] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43d44cccd80945079061c1e47fcdb161; no voters:
I20260926 01:09:05.555380 22340 leader_election.cc:288] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:09:05.555433 22345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:05.555604 22345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [term 1 LEADER]: Becoming Leader. State: Replica: 43d44cccd80945079061c1e47fcdb161, State: Running, Role: LEADER
I20260926 01:09:05.555647 22340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:05.555744 22345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d44cccd80945079061c1e47fcdb161" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } }
I20260926 01:09:05.556136 22346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43d44cccd80945079061c1e47fcdb161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d44cccd80945079061c1e47fcdb161" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } } }
I20260926 01:09:05.556210 22346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:05.557262 22347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43d44cccd80945079061c1e47fcdb161. Latest consensus state: current_term: 1 leader_uuid: "43d44cccd80945079061c1e47fcdb161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d44cccd80945079061c1e47fcdb161" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } } }
I20260926 01:09:05.557340 22347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:05.557447 22353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 01:09:05 dist-test-slave-l73v krb5kdc[22259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:05.557919 22353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:05.559803 22353 catalog_manager.cc:1347] Generated new cluster ID: 5573d15f6e2547f995d11d58979e2ff1
I20260926 01:09:05.559857 22353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:09:05.561573 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:05.551012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40037 (local address 127.0.98.62:34773)
0926 01:09:05.551215 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:05.551225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:05.551263 (+ 38us) server_negotiation.cc:407] Connection header received
0926 01:09:05.551766 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:05.551776 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:05.551843 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:05.552008 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:05.552624 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:05.553995 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:05.554950 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:05.555583 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:05.556086 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:05.558335 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:05.558339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:05.558340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:05.558352 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:05.560179 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:05.560492 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:05.560496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:05.560496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:05.560582 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:05.560750 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:05.560751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:05.560751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:05.560981 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:05.561130 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:05.561223 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 01:09:05.561355 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260926 01:09:05.562474 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:09:05.569957 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:05.563804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53029 (local address 127.0.98.62:34773)
0926 01:09:05.563852 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:05.563853 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:05.563911 (+ 58us) server_negotiation.cc:407] Connection header received
0926 01:09:05.563965 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:05.563966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:05.563977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:05.564037 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:05.564520 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:05.565297 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:05.566018 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:05.566241 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:05.566407 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:05.567583 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:05.567585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:05.567585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:05.567593 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:05.568839 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:05.569149 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:05.569150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:05.569151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:05.569203 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:05.569365 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:05.569365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:05.569366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:05.569475 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:05.569582 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:05.569783 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 01:09:05.569863 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:09:05.570756 22353 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:05.571573 22353 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:05.576227 22353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43d44cccd80945079061c1e47fcdb161: Generated new TSK 0
I20260926 01:09:05.576432 22353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:09:05.580821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 22272
2026-09-26T01:09:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:05 dist-test-slave-l73v krb5kdc[22371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:05 dist-test-slave-l73v krb5kdc[22371](info): set up 2 sockets
Sep 26 01:09:05 dist-test-slave-l73v krb5kdc[22371](info): commencing operation
krb5kdc: starting...
W20260926 01:09:06.546329 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:06 dist-test-slave-l73v krb5kdc[22371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:06.666405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44053
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:06.748311 22384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:06.748564 22384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:06.748589 22384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:06.749881 22384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:06.749981 22384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:06.750012 22384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:06.750030 22384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:06.750046 22384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:06.751796 22384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:06.752072 22384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:06.752301 22384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:06.758073 22390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:06.758374 22392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:06.758523 22384 server_base.cc:1034] running on GCE node
W20260926 01:09:06.758730 22389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:06.759109 22384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:06.759500 22384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:06.760706 22384 hybrid_clock.cc:648] HybridClock initialized: now 1790384946760692 us; error 30 us; skew 500 ppm
Sep 26 01:09:06 dist-test-slave-l73v krb5kdc[22371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:06.766690 22384 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:06.767217 22384 webserver.cc:460] Webserver started at http://127.0.98.62:35817/ using document root <none> and password file <none>
I20260926 01:09:06.767438 22384 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:06.767503 22384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:06.767644 22384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:06.768827 22384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd"
format_stamp: "Formatted at 2026-09-26 01:09:06 on dist-test-slave-l73v"
server_key: "c1173233bd00247711bdd4a6cc6ed646"
server_key_iv: "a63d3537376afe2e81ade15e22312880"
server_key_version: "encryptionkey@0"
I20260926 01:09:06.769241 22384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd"
format_stamp: "Formatted at 2026-09-26 01:09:06 on dist-test-slave-l73v"
server_key: "c1173233bd00247711bdd4a6cc6ed646"
server_key_iv: "a63d3537376afe2e81ade15e22312880"
server_key_version: "encryptionkey@0"
I20260926 01:09:06.770798 22384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:09:06.771653 22399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:06.771849 22384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:09:06.771948 22384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd"
format_stamp: "Formatted at 2026-09-26 01:09:06 on dist-test-slave-l73v"
server_key: "c1173233bd00247711bdd4a6cc6ed646"
server_key_iv: "a63d3537376afe2e81ade15e22312880"
server_key_version: "encryptionkey@0"
I20260926 01:09:06.772023 22384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:06.796813 22384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:06.799597 22384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:06.799759 22384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:06.803622 22384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44053
I20260926 01:09:06.803659 22451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44053 every 8 connection(s)
I20260926 01:09:06.804004 22384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:06.804934 22452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:06.807386 22452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd: Bootstrap starting.
I20260926 01:09:06.807981 22452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:06.808244 22452 log.cc:824] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:06.809022 22452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd: No bootstrap required, opened a new log
I20260926 01:09:06.810745 22452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44053 } }
I20260926 01:09:06.810894 22452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:06.810914 22452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f81b4c359f84c07bc77dfeb6c72a7dd, State: Initialized, Role: FOLLOWER
I20260926 01:09:06.811045 22452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44053 } }
I20260926 01:09:06.811117 22452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:09:06.811138 22452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:09:06.811187 22452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:06.811672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 22384
I20260926 01:09:06.811736 22452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44053 } }
I20260926 01:09:06.811772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:06.811836 22452 leader_election.cc:302] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f81b4c359f84c07bc77dfeb6c72a7dd; no voters:
I20260926 01:09:06.811934 392 external_mini_cluster.cc:1428] Setting key eb3d1819972a0e5d3b97fe8ce644fc6c
I20260926 01:09:06.811997 22452 leader_election.cc:288] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:09:06.812055 22455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:06.812230 22455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [term 1 LEADER]: Becoming Leader. State: Replica: 2f81b4c359f84c07bc77dfeb6c72a7dd, State: Running, Role: LEADER
I20260926 01:09:06.812273 22452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:06.812348 22455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44053 } }
I20260926 01:09:06.812722 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f81b4c359f84c07bc77dfeb6c72a7dd. Latest consensus state: current_term: 1 leader_uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44053 } } }
I20260926 01:09:06.812742 22456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f81b4c359f84c07bc77dfeb6c72a7dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44053 } } }
I20260926 01:09:06.812911 22456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:06.813076 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:06.813344 22463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:06.813935 22463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:06.816157 22463 catalog_manager.cc:1347] Generated new cluster ID: 51a423a968ed4d76b58c05bf9b16c409
I20260926 01:09:06.816231 22463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:09:06.818277 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:06.812427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58641 (local address 127.0.98.62:44053)
0926 01:09:06.812820 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:06.812830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:06.812866 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:09:06.812997 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:06.813006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:06.813073 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:06.813233 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:06.813634 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:06.815001 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:06.816089 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:06.816706 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:06.816813 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:06.817544 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:06.817548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:06.817548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:09:06.817775 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:06.817814 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:09:06.817829 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:06.817885 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:06.817973 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 01:09:06.818043 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":73,"threads_started":1}
I20260926 01:09:06.819499 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:09:06.824538 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:06.820916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52403 (local address 127.0.98.62:44053)
0926 01:09:06.820952 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:06.820953 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:06.821063 (+ 110us) server_negotiation.cc:407] Connection header received
0926 01:09:06.821145 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:06.821145 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:06.821155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:06.821180 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:06.821344 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:06.822076 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:06.823085 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:06.823380 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:06.823417 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:06.824164 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:06.824166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:06.824166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:09:06.824172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:06.824187 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:09:06.824192 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:06.824217 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:06.824418 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 01:09:06.824449 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:09:06.830793 22463 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:06.831527 22463 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:06.836961 22463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f81b4c359f84c07bc77dfeb6c72a7dd: Generated new TSK 0
I20260926 01:09:06.837189 22463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:09:06.844000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 22384
2026-09-26T01:09:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1262 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:06 dist-test-slave-l73v krb5kdc[22483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:06 dist-test-slave-l73v krb5kdc[22483](info): set up 2 sockets
Sep 26 01:09:06 dist-test-slave-l73v krb5kdc[22483](info): commencing operation
krb5kdc: starting...
W20260926 01:09:07.802441 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:07 dist-test-slave-l73v krb5kdc[22483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:07.927728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35441
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:08.007525 22496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:08.007769 22496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:08.007794 22496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:08.009028 22496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:08.009079 22496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:08.009092 22496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:08.009104 22496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:08.009114 22496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:08.010725 22496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:08.010982 22496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:08.011196 22496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:08.015868 22502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:08.015888 22504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:08.015888 22501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:08.016172 22496 server_base.cc:1034] running on GCE node
I20260926 01:09:08.016394 22496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:08.016669 22496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:08.017838 22496 hybrid_clock.cc:648] HybridClock initialized: now 1790384948017820 us; error 29 us; skew 500 ppm
Sep 26 01:09:08 dist-test-slave-l73v krb5kdc[22483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:08.020839 22496 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:08.021298 22496 webserver.cc:460] Webserver started at http://127.0.98.62:44561/ using document root <none> and password file <none>
I20260926 01:09:08.021520 22496 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:08.021564 22496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:08.021683 22496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:08.022851 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "24cdfd6d0d834073965505e6cf8a5f17"
format_stamp: "Formatted at 2026-09-26 01:09:08 on dist-test-slave-l73v"
server_key: "dedae5637b7de59c94fffeee6edacc9d"
server_key_iv: "d868a2323e0271e9e47547c62f4c7459"
server_key_version: "encryptionkey@0"
I20260926 01:09:08.023182 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "24cdfd6d0d834073965505e6cf8a5f17"
format_stamp: "Formatted at 2026-09-26 01:09:08 on dist-test-slave-l73v"
server_key: "dedae5637b7de59c94fffeee6edacc9d"
server_key_iv: "d868a2323e0271e9e47547c62f4c7459"
server_key_version: "encryptionkey@0"
I20260926 01:09:08.024457 22496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:09:08.025234 22511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:08.025455 22496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:08.025545 22496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "24cdfd6d0d834073965505e6cf8a5f17"
format_stamp: "Formatted at 2026-09-26 01:09:08 on dist-test-slave-l73v"
server_key: "dedae5637b7de59c94fffeee6edacc9d"
server_key_iv: "d868a2323e0271e9e47547c62f4c7459"
server_key_version: "encryptionkey@0"
I20260926 01:09:08.025599 22496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:08.029053 22496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:08.031217 22496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:08.031327 22496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:08.035014 22496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35441
I20260926 01:09:08.035049 22563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35441 every 8 connection(s)
I20260926 01:09:08.035374 22496 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:08.036362 22564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:08.038887 22564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17: Bootstrap starting.
I20260926 01:09:08.039502 22564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:08.039774 22564 log.cc:824] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:08.040534 22564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17: No bootstrap required, opened a new log
I20260926 01:09:08.042173 22564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cdfd6d0d834073965505e6cf8a5f17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } }
I20260926 01:09:08.042364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 22496
I20260926 01:09:08.042405 22564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:09:08.042448 22564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24cdfd6d0d834073965505e6cf8a5f17, State: Initialized, Role: FOLLOWER
I20260926 01:09:08.042482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:08.042609 22564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cdfd6d0d834073965505e6cf8a5f17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } }
I20260926 01:09:08.042671 392 external_mini_cluster.cc:1428] Setting key f4f0cf495157cfb6bed5d4c444f0e6b7
I20260926 01:09:08.042692 22564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 01:09:08.042729 22564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 01:09:08.042778 22564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:08.043465 22564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cdfd6d0d834073965505e6cf8a5f17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } }
I20260926 01:09:08.043555 22564 leader_election.cc:302] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24cdfd6d0d834073965505e6cf8a5f17; no voters:
I20260926 01:09:08.043737 22564 leader_election.cc:288] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 01:09:08.043807 22569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:08.043946 22569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [term 1 LEADER]: Becoming Leader. State: Replica: 24cdfd6d0d834073965505e6cf8a5f17, State: Running, Role: LEADER
I20260926 01:09:08.043984 22564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:08.044035 22569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cdfd6d0d834073965505e6cf8a5f17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } }
I20260926 01:09:08.044384 22571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24cdfd6d0d834073965505e6cf8a5f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cdfd6d0d834073965505e6cf8a5f17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } } }
I20260926 01:09:08.044366 22569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24cdfd6d0d834073965505e6cf8a5f17. Latest consensus state: current_term: 1 leader_uuid: "24cdfd6d0d834073965505e6cf8a5f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cdfd6d0d834073965505e6cf8a5f17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } } }
I20260926 01:09:08.044503 22571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:08.044514 22569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:08.045233 22578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:08.045732 22578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:08.045825 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:08.043143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52171 (local address 127.0.98.62:35441)
0926 01:09:08.043712 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:08.043723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:08.043759 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:09:08.043890 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:08.043897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:08.043928 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:08.044824 (+ 896us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:08.045179 (+ 355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:08.045183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:08.045183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:09:08.045187 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:08.045229 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:09:08.045245 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:08.045318 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:08.045579 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 01:09:08.045623 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":457,"threads_started":1}
I20260926 01:09:08.047188 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:09:08.047242 22578 catalog_manager.cc:1347] Generated new cluster ID: e60d10bf6f364d2586e54e4e1d487ea4
I20260926 01:09:08.047282 22578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:09:08.049789 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:08.048622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53549 (local address 127.0.98.62:35441)
0926 01:09:08.048661 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:08.048663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:08.048827 (+ 164us) server_negotiation.cc:407] Connection header received
0926 01:09:08.048903 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:08.048904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:08.048909 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:08.048933 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:08.049565 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:08.049566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:08.049567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 01:09:08.049571 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:08.049585 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 01:09:08.049589 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:08.049615 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:08.049705 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 01:09:08.049717 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:09:08.055855 22578 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:08.056048 22578 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:08.060269 22578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24cdfd6d0d834073965505e6cf8a5f17: Generated new TSK 0
I20260926 01:09:08.060447 22578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 01:09:08.067385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 22496
2026-09-26T01:09:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1223 ms)
[----------] 18 tests from AuthTokenIssuingTest (22405 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:08 dist-test-slave-l73v krb5kdc[22595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:08 dist-test-slave-l73v krb5kdc[22595](info): set up 2 sockets
Sep 26 01:09:08 dist-test-slave-l73v krb5kdc[22595](info): commencing operation
krb5kdc: starting...
W20260926 01:09:09.018303 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:09.142191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40717
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40717,127.0.98.61:44719,127.0.98.60:36907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:09.226864 22608 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:09:09.227110 22608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:09.227128 22608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:09.227150 22608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:09.228466 22608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:09.228520 22608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:09.228534 22608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:09.228545 22608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:09.228556 22608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:09.230201 22608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40717,127.0.98.61:44719,127.0.98.60:36907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:09.230505 22608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:09.230738 22608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:09:09.235826 22608 server_base.cc:1034] running on GCE node
W20260926 01:09:09.235759 22616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:09.235903 22614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:09.235764 22613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:09.236176 22608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:09.236493 22608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:09.237607 22608 hybrid_clock.cc:648] HybridClock initialized: now 1790384949237594 us; error 30 us; skew 500 ppm
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:09.240151 22608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:09.240607 22608 webserver.cc:460] Webserver started at http://127.0.98.62:37383/ using document root <none> and password file <none>
I20260926 01:09:09.240826 22608 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:09.240870 22608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:09.241001 22608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:09.242122 22608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4"
format_stamp: "Formatted at 2026-09-26 01:09:09 on dist-test-slave-l73v"
server_key: "dc89c86e816ee8b419eb63a02a877a11"
server_key_iv: "ecac89273396f47d7264d9b3588fd7cd"
server_key_version: "encryptionkey@0"
I20260926 01:09:09.242482 22608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4"
format_stamp: "Formatted at 2026-09-26 01:09:09 on dist-test-slave-l73v"
server_key: "dc89c86e816ee8b419eb63a02a877a11"
server_key_iv: "ecac89273396f47d7264d9b3588fd7cd"
server_key_version: "encryptionkey@0"
I20260926 01:09:09.243803 22608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:09.244565 22623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:09.244721 22608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:09:09.244837 22608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4"
format_stamp: "Formatted at 2026-09-26 01:09:09 on dist-test-slave-l73v"
server_key: "dc89c86e816ee8b419eb63a02a877a11"
server_key_iv: "ecac89273396f47d7264d9b3588fd7cd"
server_key_version: "encryptionkey@0"
I20260926 01:09:09.244906 22608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:09.255676 22608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:09.257823 22608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:09.257944 22608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:09.261912 22608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40717
I20260926 01:09:09.261993 22675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40717 every 8 connection(s)
I20260926 01:09:09.262310 22608 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:09.263355 22676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:09.264950 22676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:09:09.268187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 22608
I20260926 01:09:09.268312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:09.268498 392 external_mini_cluster.cc:1428] Setting key f6a3e244ab44c29e33c1498a00ad503b
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:09.276307 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.265621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40717 (local address 127.0.0.1:47604)
0926 01:09:09.265808 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.265846 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:09.266349 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:09.266701 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:09.266704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:09.266979 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:09.267144 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.267148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.268222 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.268223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.269251 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.269254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.269805 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.269807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.269890 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.270508 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:09.270518 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:09.272066 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:09.274310 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.274319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.274321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.274689 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.275091 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.275092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.275092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.275194 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.275787 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:09.275794 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:09.275884 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 01:09:09.275963 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 01:09:09.276117 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260926 01:09:09.276621 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.265786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.0.98.62:40717)
0926 01:09:09.265942 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.265945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.265974 (+ 29us) server_negotiation.cc:407] Connection header received
0926 01:09:09.266547 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.266550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.266569 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.266642 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.267269 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.268103 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.269341 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.269665 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.269787 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.272170 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.272180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.272180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.272190 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.274190 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.274787 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.274791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.274791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.274998 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.275271 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.275272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.275272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.275681 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.275794 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.276043 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.276208 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":129,"thread_start_us":56,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260926 01:09:09.277328 22676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:09:09.277684 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.277563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44719 (local address 127.0.0.1:56108)
0926 01:09:09.277611 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.277662 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:09:09.277863 22624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44719: connect: Connection refused (error 111)
I20260926 01:09:09.278287 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.278193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44719 (local address 127.0.0.1:56114)
0926 01:09:09.278240 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.278266 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 01:09:09.278414 22676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44719: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 01:09:09.293128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44719
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40717,127.0.98.61:44719,127.0.98.60:36907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:09:09.327687 22676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } attempt: 1
I20260926 01:09:09.328310 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.328130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44719 (local address 127.0.0.1:56120)
0926 01:09:09.328231 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.328277 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 01:09:09.328743 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.328608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44719 (local address 127.0.0.1:56128)
0926 01:09:09.328696 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.328721 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 01:09:09.328845 22676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44719: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44719: connect: Connection refused (error 111)
W20260926 01:09:09.373349 22683 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:09:09.373580 22683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:09.373597 22683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:09.373620 22683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:09.374900 22683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:09.374953 22683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:09.374966 22683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:09.374979 22683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:09.374991 22683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:09.376618 22683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40717,127.0.98.61:44719,127.0.98.60:36907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:09.376892 22683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:09.377126 22683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:09.382332 22688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:09.382333 22689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:09.382529 22683 server_base.cc:1034] running on GCE node
W20260926 01:09:09.382688 22691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:09.382936 22683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:09.383222 22683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:09.384385 22683 hybrid_clock.cc:648] HybridClock initialized: now 1790384949384368 us; error 29 us; skew 500 ppm
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:09.387127 22683 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 01:09:09.387603 22683 webserver.cc:460] Webserver started at http://127.0.98.61:37715/ using document root <none> and password file <none>
I20260926 01:09:09.387806 22683 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:09.387854 22683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:09.387984 22683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:09.389050 22683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/data/instance:
uuid: "fec0ee6088044ec9acbb53842beef696"
format_stamp: "Formatted at 2026-09-26 01:09:09 on dist-test-slave-l73v"
server_key: "b3b149ba3d6aa515956345103a359943"
server_key_iv: "4ddd8b49423d71ad6c02431d1d23707b"
server_key_version: "encryptionkey@0"
I20260926 01:09:09.389398 22683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/wal/instance:
uuid: "fec0ee6088044ec9acbb53842beef696"
format_stamp: "Formatted at 2026-09-26 01:09:09 on dist-test-slave-l73v"
server_key: "b3b149ba3d6aa515956345103a359943"
server_key_iv: "4ddd8b49423d71ad6c02431d1d23707b"
server_key_version: "encryptionkey@0"
I20260926 01:09:09.390870 22683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:09:09.391631 22698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:09.391788 22683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 01:09:09.391871 22683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/wal
uuid: "fec0ee6088044ec9acbb53842beef696"
format_stamp: "Formatted at 2026-09-26 01:09:09 on dist-test-slave-l73v"
server_key: "b3b149ba3d6aa515956345103a359943"
server_key_iv: "4ddd8b49423d71ad6c02431d1d23707b"
server_key_version: "encryptionkey@0"
I20260926 01:09:09.391935 22683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:09.397006 22676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } attempt: 2
I20260926 01:09:09.397478 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.397334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44719 (local address 127.0.0.1:56140)
0926 01:09:09.397393 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.397439 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 01:09:09.397882 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.397793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44719 (local address 127.0.0.1:56152)
0926 01:09:09.397832 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.397859 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 01:09:09.397966 22676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44719: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44719: connect: Connection refused (error 111)
I20260926 01:09:09.411676 22683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:09.413771 22683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:09.413887 22683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:09.417881 22683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44719
I20260926 01:09:09.417915 22750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44719 every 8 connection(s)
I20260926 01:09:09.418260 22683 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
I20260926 01:09:09.419250 22751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:09.421132 22751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:09.428256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 22683
I20260926 01:09:09.428354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-1/wal/instance
I20260926 01:09:09.428511 392 external_mini_cluster.cc:1428] Setting key 999b639017408f3fbf496f3a101fb369
I20260926 01:09:09.430071 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.421944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.0.98.62:40717)
0926 01:09:09.422009 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.422011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.422174 (+ 163us) server_negotiation.cc:407] Connection header received
0926 01:09:09.422679 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.422680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.422692 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.422753 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.423342 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.423821 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.424869 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.425073 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.425129 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.427266 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.427268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.427269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.427275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.428733 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.429162 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.429163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.429163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.429226 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.429444 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.429445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.429445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.429568 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.429628 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.429888 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.429977 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 01:09:09.430272 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.421889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40717 (local address 127.0.0.1:47612)
0926 01:09:09.422053 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.422095 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:09.422602 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:09.422808 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:09.422811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:09.423057 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:09.423269 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.423273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.423933 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.423935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.424775 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.424778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.425167 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.425168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.425275 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.425753 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:09.425762 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:09.427173 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:09.428809 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.428819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.428821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.429045 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.429281 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.429282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.429283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.429388 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.429630 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:09.429637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:09.429773 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 01:09:09.429863 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 01:09:09.430032 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260926 01:09:09.430922 22751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 01:09:09.437949 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.431191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44719 (local address 127.0.0.1:56156)
0926 01:09:09.431232 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.431242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:09.431278 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:09.431525 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:09.431526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:09.431711 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:09.431760 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.431762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.432584 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.432585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.433083 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.433084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.433510 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.433511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.433579 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.433958 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:09.433964 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:09.435310 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:09.437060 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.437061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.437062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.437223 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.437391 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.437392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.437392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.437449 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.437686 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:09.437687 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:09.437725 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 01:09:09.437771 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 01:09:09.437869 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 01:09:09.438055 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.431268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.0.98.61:44719)
0926 01:09:09.431417 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.431421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.431437 (+ 16us) server_negotiation.cc:407] Connection header received
0926 01:09:09.431451 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.431452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.431462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.431514 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.431825 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.432509 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.433149 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.433449 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.433507 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.435396 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.435406 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.435406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.435414 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.436982 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.437287 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.437295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.437295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.437341 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.437487 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.437487 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.437487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.437618 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.437673 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.437795 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.437932 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260926 01:09:09.438881 22751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:09:09.439217 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.439108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57022)
0926 01:09:09.439147 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.439191 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 01:09:09.439332 22701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
I20260926 01:09:09.439739 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.439643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57036)
0926 01:09:09.439683 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.439715 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:09:09.439838 22751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 01:09:09.452643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36907
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40717,127.0.98.61:44719,127.0.98.60:36907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:09:09.489128 22676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } attempt: 3
I20260926 01:09:09.489076 22751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } attempt: 1
I20260926 01:09:09.489558 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.489430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57048)
0926 01:09:09.489489 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.489523 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 01:09:09.489948 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.489839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57056)
0926 01:09:09.489890 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.489920 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 01:09:09.490106 22751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:09:09.496140 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.489617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.0.98.61:44719)
0926 01:09:09.489663 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.489664 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.489672 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:09:09.489697 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.489698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.489708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.489753 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.490112 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.490695 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.491379 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.491596 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.491658 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.493651 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.493652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.493653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.493659 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.495000 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.495457 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.495458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.495458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.495516 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.495694 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.495695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.495695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.495806 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.495869 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.496008 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.496075 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:09:09.496212 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.489451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44719 (local address 127.0.0.1:56160)
0926 01:09:09.489496 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.489506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:09.489546 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:09.489782 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:09.489784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:09.489955 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:09.490035 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.490038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.490785 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.490787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.491303 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.491305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.491657 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.491658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.491739 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.492204 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:09.492208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:09.493570 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:09.495079 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.495080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.495080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.495364 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.495581 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.495581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.495582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.495645 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.495865 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:09.495866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:09.495918 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 01:09:09.495991 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 01:09:09.496090 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:09:09.496687 22676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:09:09.497013 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.496905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57062)
0926 01:09:09.496948 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.496984 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 01:09:09.497320 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.497231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57076)
0926 01:09:09.497268 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.497301 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 01:09:09.497416 22676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
I20260926 01:09:09.528586 22676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } attempt: 1
I20260926 01:09:09.529160 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.528975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57090)
0926 01:09:09.529067 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.529114 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 01:09:09.529537 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.529454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57104)
0926 01:09:09.529494 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.529519 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:09:09.529631 22676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
W20260926 01:09:09.534814 22758 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:09:09.535027 22758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:09.535043 22758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:09.535065 22758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:09.536196 22758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:09.536231 22758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:09.536243 22758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:09.536255 22758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:09.536266 22758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:09.537693 22758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40717,127.0.98.61:44719,127.0.98.60:36907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:09.537900 22758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:09.538084 22758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:09.542893 22763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:09.542924 22766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:09.543073 22758 server_base.cc:1034] running on GCE node
W20260926 01:09:09.542939 22764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:09.543435 22758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:09.543735 22758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:09.544902 22758 hybrid_clock.cc:648] HybridClock initialized: now 1790384949544891 us; error 28 us; skew 500 ppm
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:09.547365 22758 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 01:09:09.547930 22758 webserver.cc:460] Webserver started at http://127.0.98.60:41771/ using document root <none> and password file <none>
I20260926 01:09:09.548175 22758 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:09.548230 22758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:09.548327 22758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:09.549342 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/data/instance:
uuid: "17666fd0341d4378af2ab86567e801a6"
format_stamp: "Formatted at 2026-09-26 01:09:09 on dist-test-slave-l73v"
server_key: "78c5fdcb108c32e47ee63e0270fd1cbc"
server_key_iv: "0a2bd54ebc42801216ac3a2d4d01695e"
server_key_version: "encryptionkey@0"
I20260926 01:09:09.549626 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/wal/instance:
uuid: "17666fd0341d4378af2ab86567e801a6"
format_stamp: "Formatted at 2026-09-26 01:09:09 on dist-test-slave-l73v"
server_key: "78c5fdcb108c32e47ee63e0270fd1cbc"
server_key_iv: "0a2bd54ebc42801216ac3a2d4d01695e"
server_key_version: "encryptionkey@0"
I20260926 01:09:09.550884 22758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 01:09:09.551594 22773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:09.551743 22758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 01:09:09.551822 22758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/wal
uuid: "17666fd0341d4378af2ab86567e801a6"
format_stamp: "Formatted at 2026-09-26 01:09:09 on dist-test-slave-l73v"
server_key: "78c5fdcb108c32e47ee63e0270fd1cbc"
server_key_iv: "0a2bd54ebc42801216ac3a2d4d01695e"
server_key_version: "encryptionkey@0"
I20260926 01:09:09.551887 22758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:09.558279 22751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } attempt: 2
I20260926 01:09:09.558712 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.558574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57110)
0926 01:09:09.558633 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.558675 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 01:09:09.559063 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.558995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57118)
0926 01:09:09.559027 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.559048 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 01:09:09.559194 22751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36907: connect: Connection refused (error 111)
I20260926 01:09:09.572551 22758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:09.574766 22758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:09.574887 22758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:09.578778 22758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36907
I20260926 01:09:09.578810 22825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36907 every 8 connection(s)
I20260926 01:09:09.579190 22758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
I20260926 01:09:09.580149 22826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:09.581689 22826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:09.587760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 22758
I20260926 01:09:09.587875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790384680280236-392-0/minicluster-data/master-2/wal/instance
I20260926 01:09:09.588081 392 external_mini_cluster.cc:1428] Setting key 52efd7e13aa618ce54cc14285ad73696
I20260926 01:09:09.591152 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.582446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.0.98.62:40717)
0926 01:09:09.582501 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.582503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.582672 (+ 169us) server_negotiation.cc:407] Connection header received
0926 01:09:09.583153 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.583154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.583165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.583204 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.583830 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.584303 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.585333 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.585578 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.585641 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.587962 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.587964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.587964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.587972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.589666 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.590100 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.590102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.590102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.590168 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.590427 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.590428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.590428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.590569 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.590636 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.590945 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.591038 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 01:09:09.591285 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.582363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40717 (local address 127.0.0.1:47628)
0926 01:09:09.582544 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.582586 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:09.583052 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:09.583272 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:09.583275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:09.583527 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:09.583748 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.583752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.584398 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.584399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.585231 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.585233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.585663 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.585664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.585772 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.586275 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:09.586284 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:09.587858 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:09.589755 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.589763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.589765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.590017 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.590229 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.590230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.590230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.590345 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.590631 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:09.590638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:09.590787 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 01:09:09.590878 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 01:09:09.591060 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260926 01:09:09.592120 22826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:09:09.596989 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.588477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.0.98.60:36907)
0926 01:09:09.588702 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.588707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.588737 (+ 30us) server_negotiation.cc:407] Connection header received
0926 01:09:09.588760 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.588762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.588775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.588839 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.589405 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.590118 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.590885 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.591207 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.591333 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.593699 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.593708 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.593708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.593715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.595728 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.596128 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.596133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.596133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.596194 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.596427 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.596428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.596429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.596596 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.596719 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.596809 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.596880 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":138,"threads_started":1}
I20260926 01:09:09.599442 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.592533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.0.98.61:44719)
0926 01:09:09.592580 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.592582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.592594 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:09:09.592631 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.592632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.592642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.592698 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.592973 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.593580 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.594528 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.594796 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.594869 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.596706 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.596708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.596708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.596716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.598363 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.598735 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.598737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.598737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.598803 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.598984 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.598985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.598985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.599129 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.599192 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.599307 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.599375 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 01:09:09.599465 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.592414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44719 (local address 127.0.0.1:56170)
0926 01:09:09.592460 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.592471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:09.592570 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:09.592703 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:09.592703 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:09.592844 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:09.592897 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.592899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.593689 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.593690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.594445 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.594447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.594870 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.594871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.594946 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.595298 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:09.595303 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:09.596620 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:09.598484 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.598485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.598485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.598670 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.598889 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.598889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.598889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.598937 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.599183 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:09.599184 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:09.599218 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 01:09:09.599268 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 01:09:09.599365 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:09:09.599875 22826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:09:09.604782 22676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } attempt: 2
I20260926 01:09:09.605340 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.597835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.0.98.62:40717)
0926 01:09:09.597872 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.597874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.597882 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:09:09.597952 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.597953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.597960 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.598002 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.598600 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.599218 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.600099 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.600326 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.600384 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.602554 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.602555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.602555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.602570 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.604113 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.604432 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.604434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.604434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.604486 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.604651 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.604652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.604653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.604798 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.604955 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.605166 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.605250 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:09:09.606185 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.600130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57134)
0926 01:09:09.600171 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.600180 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:09.600212 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:09.600310 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:09.600310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:09.600434 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:09.600462 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.600463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.601177 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.601178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.601640 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.601641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.601964 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.601965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.602007 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.602372 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:09.602392 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:09.603701 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:09.605204 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.605204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.605205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.605363 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.605572 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.605573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.605574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.605645 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.605874 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:09.605875 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:09.605922 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 01:09:09.605982 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 01:09:09.606079 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 01:09:09.606223 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.600186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.0.98.60:36907)
0926 01:09:09.600220 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.600222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.600230 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:09:09.600247 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.600248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.600256 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.600304 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.600519 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.601105 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.601703 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.601911 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.601963 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.603784 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.603785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.603785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.603796 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.605123 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.605451 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.605452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.605452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.605507 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.605709 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.605709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.605710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.605821 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.605875 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.606016 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.606123 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":21}
I20260926 01:09:09.609020 22826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6: Bootstrap starting.
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:09:09.609714 22826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:09.610023 22826 log.cc:824] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6: Log is configured to *not* fsync() on all Append() calls
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:09:09.610985 22826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6: No bootstrap required, opened a new log
I20260926 01:09:09.612776 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.605025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57148)
0926 01:09:09.605076 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.605085 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:09.605117 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:09.605363 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:09.605364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:09.605503 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:09.605540 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.605541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.606718 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.606719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.607325 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.607328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.607966 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.607967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.608055 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.608499 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:09.608504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:09.609996 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:09.611717 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.611719 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.611719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.611885 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.612070 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.612071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.612071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.612136 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.612447 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:09.612448 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:09.612497 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 01:09:09.612558 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 01:09:09.612675 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 01:09:09.612807 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.605061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.0.98.60:36907)
0926 01:09:09.605237 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.605239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.605245 (+ 6us) server_negotiation.cc:407] Connection header received
0926 01:09:09.605262 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.605263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.605272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.605334 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.605710 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.606629 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.607590 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.607887 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.607951 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.610097 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.610099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.610099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.610107 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.611635 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.611947 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.611948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.611949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.612011 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.612207 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.612208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.612208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.612380 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.612443 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.612603 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.612678 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":114,"threads_started":1}
I20260926 01:09:09.613286 22826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } }
I20260926 01:09:09.613554 22826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17666fd0341d4378af2ab86567e801a6, State: Initialized, Role: FOLLOWER
I20260926 01:09:09.613651 22826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } }
I20260926 01:09:09.613938 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } } }
I20260926 01:09:09.614007 22826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:09.614570 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.606201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.0.98.61:44719)
0926 01:09:09.606236 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.606238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.606307 (+ 69us) server_negotiation.cc:407] Connection header received
0926 01:09:09.606412 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.606413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.606424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.606473 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.607050 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.607555 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.608365 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.608634 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.608668 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.611188 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.611190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.611190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.611200 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.613046 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.613542 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.613544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.613544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.613614 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.613807 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.613807 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.613808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.613950 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.614115 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.614360 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.614473 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:09:09.614696 22676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4: Bootstrap starting.
I20260926 01:09:09.615454 22676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:09.615779 22676 log.cc:824] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:09.616304 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 01:09:09.616477 22844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:09:09.616520 22844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:09:09.614084 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:09.617724 22676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4: No bootstrap required, opened a new log
I20260926 01:09:09.617225 22650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47632
I20260926 01:09:09.616340 22758 master_runner.cc:186] Error getting master registration for 127.0.98.62:40717: OK, instance_id { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" instance_seqno: 1790384949260721 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:09:09.618635 22758 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:09:09.619009 22676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } }
I20260926 01:09:09.619128 22676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd70ddd43604ab9b48bc4faf7fcbae4, State: Initialized, Role: FOLLOWER
I20260926 01:09:09.619191 22676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } }
I20260926 01:09:09.619483 22676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:09.619478 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } } }
I20260926 01:09:09.619580 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:09.620901 22650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 01:09:09.621112 22650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:09:09.621361 22858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:09:09.621400 22858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:09:09.622061 22650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } }
I20260926 01:09:09.622459 22650 leader_election.cc:288] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [CANDIDATE]: Term 1 election: Requested vote from peers fec0ee6088044ec9acbb53842beef696 (127.0.98.61:44719), 17666fd0341d4378af2ab86567e801a6 (127.0.98.60:36907)
I20260926 01:09:09.622838 22800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "17666fd0341d4378af2ab86567e801a6"
I20260926 01:09:09.622961 22800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:09.623867 22800 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd70ddd43604ab9b48bc4faf7fcbae4 in term 1.
I20260926 01:09:09.624890 22626 leader_election.cc:302] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17666fd0341d4378af2ab86567e801a6, 2cd70ddd43604ab9b48bc4faf7fcbae4; no voters:
I20260926 01:09:09.625005 22847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:09.625270 22847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd70ddd43604ab9b48bc4faf7fcbae4, State: Running, Role: LEADER
I20260926 01:09:09.625378 22847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } }
I20260926 01:09:09.625811 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd70ddd43604ab9b48bc4faf7fcbae4. Latest consensus state: current_term: 1 leader_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } } }
I20260926 01:09:09.625895 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:09.623466 22608 master_runner.cc:186] Error getting master registration for 127.0.98.61:44719: OK, instance_id { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" instance_seqno: 1790384949416764 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:09:09.626097 22608 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:09:09.626963 22866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:09.627184 22866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:09.629037 22800 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [term 1 FOLLOWER]: Refusing update from remote peer 2cd70ddd43604ab9b48bc4faf7fcbae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:09:09.629369 22847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:09:09.630729 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd70ddd43604ab9b48bc4faf7fcbae4. Latest consensus state: current_term: 1 leader_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } } }
I20260926 01:09:09.630831 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:09.640992 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.628550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.0.98.60:36907)
0926 01:09:09.628593 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.628594 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.630040 (+ 1446us) server_negotiation.cc:407] Connection header received
0926 01:09:09.630198 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.630200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.630223 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.630251 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.631457 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.632182 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.633331 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.633760 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.633815 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.636691 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.636693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.636693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.636705 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.638624 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.639772 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.639774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.639774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.639842 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.640323 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.640325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.640325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.640481 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.640580 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.640802 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.640890 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 01:09:09.645503 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.628310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.0.98.62:40717)
0926 01:09:09.628344 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.628346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.631579 (+ 3233us) server_negotiation.cc:407] Connection header received
0926 01:09:09.631713 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.631714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.631724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.631751 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.633762 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.634497 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.637497 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.637787 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.637830 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.639140 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.639142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.639142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.639149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.640830 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.644479 (+ 3649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.644482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.644482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.644556 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.644775 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.644776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.644777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.644917 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.645015 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.645328 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.645413 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 01:09:09.645746 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } } }
I20260926 01:09:09.645843 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:09.646345 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } } }
I20260926 01:09:09.646476 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:09.646657 22867 mvcc.cc:205] Tried to move back new op lower bound from 7333416753678065664 to 7333416753666834432. Current Snapshot: MvccSnapshot[applied={T|T < 7333416753678065664}]
I20260926 01:09:09.646867 22871 mvcc.cc:205] Tried to move back new op lower bound from 7333416753678065664 to 7333416753666834432. Current Snapshot: MvccSnapshot[applied={T|T < 7333416753678065664}]
I20260926 01:09:09.647912 22866 catalog_manager.cc:1347] Generated new cluster ID: 78cd821ca01f463580a5e88eb62eed62
I20260926 01:09:09.647958 22866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:09:09.650344 22751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } attempt: 3
I20260926 01:09:09.651757 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.628100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.0.98.61:44719)
0926 01:09:09.628144 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.628147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.636771 (+ 8624us) server_negotiation.cc:407] Connection header received
0926 01:09:09.636865 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.636866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.636878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.636907 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.641448 (+ 4541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.642143 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.643145 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.643898 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.643970 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.648340 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.648342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.648342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.648351 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.650045 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.650757 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.650759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.650759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.650831 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.651066 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.651067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.651067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.651203 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.651296 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.651593 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.651666 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 01:09:09 dist-test-slave-l73v krb5kdc[22595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:09:09.658330 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.650636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36907 (local address 127.0.0.1:57166)
0926 01:09:09.650729 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:09.650739 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:09.650775 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:09.651018 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:09.651019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:09.651190 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:09.651234 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.651236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.652160 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.652161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.652947 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:09.652949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.653466 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.653467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:09.653522 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.653960 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:09.653965 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:09.655462 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:09.657290 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.657291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.657292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.657493 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.657689 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:09.657690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:09.657690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:09.657744 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:09.658004 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:09.658005 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:09.658049 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:09:09.658120 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 01:09:09.658224 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 01:09:09.658694 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:09.650845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.0.98.60:36907)
0926 01:09:09.650890 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.650892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.650899 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:09:09.650920 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.650921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.650929 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.650973 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.651315 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.652053 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.653038 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:09.653353 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:09.653411 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:09.655561 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:09.655562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:09.655563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:09.655571 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:09.657192 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.657566 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.657567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.657568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.657626 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:09.657802 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:09.657802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:09.657802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:09.657928 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:09.657981 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:09.658526 (+ 545us) server_negotiation.cc:299] Negotiation successful
0926 01:09:09.658603 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 01:09:09.660254 22751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696: Bootstrap starting.
I20260926 01:09:09.661020 22751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:09.661335 22751 log.cc:824] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:09.662261 22751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696: No bootstrap required, opened a new log
I20260926 01:09:09.664295 22751 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } }
I20260926 01:09:09.664443 22751 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec0ee6088044ec9acbb53842beef696, State: Initialized, Role: FOLLOWER
I20260926 01:09:09.664906 22751 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } }
I20260926 01:09:09.666296 22866 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:09.666206 22725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fec0ee6088044ec9acbb53842beef696"
I20260926 01:09:09.666271 22724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:09.666981 22866 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:09.667464 22724 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [term 1 FOLLOWER]: Refusing update from remote peer 2cd70ddd43604ab9b48bc4faf7fcbae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:09:09.667771 22864 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:09:09.668445 22725 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd70ddd43604ab9b48bc4faf7fcbae4 in term 1.
I20260926 01:09:09.666149 22751 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:09.674713 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } } }
I20260926 01:09:09.674902 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:09.674983 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd70ddd43604ab9b48bc4faf7fcbae4. Latest consensus state: current_term: 1 leader_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } } }
I20260926 01:09:09.675063 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:09.675123 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } } }
I20260926 01:09:09.675205 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:09.676285 22878 mvcc.cc:205] Tried to move back new op lower bound from 7333416753825234944 to 7333416753666834432. Current Snapshot: MvccSnapshot[applied={T|T < 7333416753678065664}]
W20260926 01:09:09.676985 22891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:09:09.677081 22891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:09:09.680047 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd70ddd43604ab9b48bc4faf7fcbae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40717 } } peers { permanent_uuid: "fec0ee6088044ec9acbb53842beef696" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44719 } } peers { permanent_uuid: "17666fd0341d4378af2ab86567e801a6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36907 } } }
I20260926 01:09:09.680164 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:09.683054 22866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cd70ddd43604ab9b48bc4faf7fcbae4: Generated new TSK 0
I20260926 01:09:09.683284 22866 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:09:09.711170 22831 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.709386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.0.98.60:36907)
0926 01:09:09.709455 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.709458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.710442 (+ 984us) server_negotiation.cc:407] Connection header received
0926 01:09:09.710540 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.710542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.710556 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.710587 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.711093 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57180: BlockingRecv error: recv got EOF from 127.0.0.1:57180 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 01:09:09.711946 22679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.709773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.0.98.62:40717)
0926 01:09:09.709815 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.709817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.711415 (+ 1598us) server_negotiation.cc:407] Connection header received
0926 01:09:09.711509 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.711510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.711524 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.711555 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.711875 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47644: BlockingRecv error: recv got EOF from 127.0.0.1:47644 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:09:09.712678 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.710147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.0.98.61:44719)
0926 01:09:09.710188 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.710190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.712186 (+ 1996us) server_negotiation.cc:407] Connection header received
0926 01:09:09.712279 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.712280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.712291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.712318 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.712613 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56208: BlockingRecv error: recv got EOF from 127.0.0.1:56208 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:09:09.715407 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.713984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.0.98.61:44719)
0926 01:09:09.714020 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.714022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.714785 (+ 763us) server_negotiation.cc:407] Connection header received
0926 01:09:09.714912 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.714914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.714933 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.714962 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.715353 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56224: BlockingRecv error: recv got EOF from 127.0.0.1:56224 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 01:09:09.718683 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.717792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.0.98.61:44719)
0926 01:09:09.717828 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.717830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.718053 (+ 223us) server_negotiation.cc:407] Connection header received
0926 01:09:09.718148 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.718149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.718160 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.718214 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.718628 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56240: BlockingRecv error: recv got EOF from 127.0.0.1:56240 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 01:09:09.723506 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.722658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.0.98.61:44719)
0926 01:09:09.722713 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.722715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.722943 (+ 228us) server_negotiation.cc:407] Connection header received
0926 01:09:09.723050 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.723052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.723063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.723093 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.723447 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56252: BlockingRecv error: recv got EOF from 127.0.0.1:56252 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 01:09:09.729693 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.728831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.0.98.61:44719)
0926 01:09:09.728873 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.728875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.729116 (+ 241us) server_negotiation.cc:407] Connection header received
0926 01:09:09.729215 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.729217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.729228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.729257 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.729628 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56264: BlockingRecv error: recv got EOF from 127.0.0.1:56264 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 01:09:09.739977 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.739104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.0.98.61:44719)
0926 01:09:09.739147 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.739149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.739389 (+ 240us) server_negotiation.cc:407] Connection header received
0926 01:09:09.739491 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.739492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.739504 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.739532 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.739910 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56270: BlockingRecv error: recv got EOF from 127.0.0.1:56270 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 01:09:09.758344 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.757420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.0.98.61:44719)
0926 01:09:09.757463 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.757465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.757726 (+ 261us) server_negotiation.cc:407] Connection header received
0926 01:09:09.757829 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.757831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.757843 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.757872 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.758283 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56284: BlockingRecv error: recv got EOF from 127.0.0.1:56284 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:09:09.792711 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.791746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.0.98.61:44719)
0926 01:09:09.791801 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.791803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.792085 (+ 282us) server_negotiation.cc:407] Connection header received
0926 01:09:09.792193 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.792194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.792207 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.792237 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.792650 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56300: BlockingRecv error: recv got EOF from 127.0.0.1:56300 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260926 01:09:09.858868 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.858119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.0.98.61:44719)
0926 01:09:09.858213 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.858215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.858229 (+ 14us) server_negotiation.cc:407] Connection header received
0926 01:09:09.858289 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.858289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.858301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.858368 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.858792 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56302: BlockingRecv error: recv got EOF from 127.0.0.1:56302 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 01:09:09.988670 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:09.988030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.0.98.61:44719)
0926 01:09:09.988080 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:09.988082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:09.988139 (+ 57us) server_negotiation.cc:407] Connection header received
0926 01:09:09.988191 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:09.988192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:09.988202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:09.988251 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:09.988606 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56314: BlockingRecv error: recv got EOF from 127.0.0.1:56314 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 01:09:10.246521 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:10.245842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.0.98.61:44719)
0926 01:09:10.245977 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:10.245979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:10.245990 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:09:10.246016 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:10.246017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:10.246027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:10.246084 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:10.246460 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56324: BlockingRecv error: recv got EOF from 127.0.0.1:56324 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 01:09:10.616933 22844 catalog_manager.cc:1259] Loaded cluster ID: 78cd821ca01f463580a5e88eb62eed62
I20260926 01:09:10.617014 22844 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6: loading cluster ID for follower catalog manager: success
I20260926 01:09:10.617867 22844 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6: acquiring CA information for follower catalog manager: success
I20260926 01:09:10.618168 22844 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 17666fd0341d4378af2ab86567e801a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:09:10.677601 22891 catalog_manager.cc:1259] Loaded cluster ID: 78cd821ca01f463580a5e88eb62eed62
I20260926 01:09:10.677695 22891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696: loading cluster ID for follower catalog manager: success
I20260926 01:09:10.678622 22891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696: acquiring CA information for follower catalog manager: success
I20260926 01:09:10.678931 22891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fec0ee6088044ec9acbb53842beef696: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:09:10.761941 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:10.759610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.0.98.61:44719)
0926 01:09:10.759703 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:10.759704 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:10.759758 (+ 54us) server_negotiation.cc:407] Connection header received
0926 01:09:10.759859 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:10.759860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:10.759905 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:10.759972 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:09:10.760069 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:10.760571 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:10.761306 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:10.761503 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:10.761551 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:10.761653 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:09:10.761758 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:09:10.761808 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:10.761840 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 01:09:10.761856 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":45,"threads_started":1}
I20260926 01:09:10.765596 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:10.763567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.0.98.60:36907)
0926 01:09:10.763664 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:10.763666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:10.763702 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:09:10.763767 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:10.763768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:10.763800 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:10.763860 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:09:10.763923 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:10.764386 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:10.765030 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:10.765206 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:10.765279 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:10.765345 (+ 66us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:09:10.765439 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:09:10.765490 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:10.765501 (+ 11us) server_negotiation.cc:299] Negotiation successful
0926 01:09:10.765516 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":44,"threads_started":1}
I20260926 01:09:10.770263 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:10.767253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.98.62:40717)
0926 01:09:10.767354 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:10.767356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:10.767466 (+ 110us) server_negotiation.cc:407] Connection header received
0926 01:09:10.767515 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:10.767516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:10.767564 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:10.767639 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:09:10.767729 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:10.768471 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:10.769445 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:10.769643 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:10.769694 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:10.769858 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:09:10.769962 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:09:10.770021 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:10.770074 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 01:09:10.770098 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20260926 01:09:10.770625 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:10.767351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.0.98.61:44719)
0926 01:09:10.767395 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:10.767397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:10.767567 (+ 170us) server_negotiation.cc:407] Connection header received
0926 01:09:10.767620 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:10.767621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:10.767632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:10.767676 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:09:10.767812 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:10.768266 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:10.769797 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:10.770068 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:10.770140 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:10.770337 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:09:10.770373 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:09:10.770423 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:10.770532 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 01:09:10.770561 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:09:10.770691 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:10.767248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.0.98.60:36907)
0926 01:09:10.767312 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:10.767314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:10.767324 (+ 10us) server_negotiation.cc:407] Connection header received
0926 01:09:10.767374 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:10.767375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:10.767386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:10.767443 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:09:10.767533 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:10.768257 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:10.769975 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:10.770246 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:10.770307 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:10.770466 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:09:10.770497 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:09:10.770521 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:10.770598 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 01:09:10.770620 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 01:09:10.772040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 22608
I20260926 01:09:10.777141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 22683
I20260926 01:09:10.781420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 22758
2026-09-26T01:09:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2715 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 01:09:10 dist-test-slave-l73v krb5kdc[22977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 01:09:10 dist-test-slave-l73v krb5kdc[22977](info): set up 2 sockets
Sep 26 01:09:10 dist-test-slave-l73v krb5kdc[22977](info): commencing operation
krb5kdc: starting...
W20260926 01:09:11.738511 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/test-user.keytab.
Sep 26 01:09:11 dist-test-slave-l73v krb5kdc[22977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T01:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:09:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 01:09:11.857964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44297,127.0.98.61:42523,127.0.98.60:36197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 01:09:11.940975 22990 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:09:11.941236 22990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:11.941262 22990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:11.941282 22990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:11.942620 22990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:11.942669 22990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:11.942682 22990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:11.942694 22990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:11.942705 22990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:11.944370 22990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44297,127.0.98.61:42523,127.0.98.60:36197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:11.944646 22990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:11.944904 22990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:09:11.949962 22990 server_base.cc:1034] running on GCE node
W20260926 01:09:11.949982 22996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:11.949926 22998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:11.949932 22995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:11.950471 22990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:11.950784 22990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:11.951957 22990 hybrid_clock.cc:648] HybridClock initialized: now 1790384951951936 us; error 42 us; skew 500 ppm
Sep 26 01:09:11 dist-test-slave-l73v krb5kdc[22977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:11.954600 22990 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 01:09:11.955090 22990 webserver.cc:460] Webserver started at http://127.0.98.62:45485/ using document root <none> and password file <none>
I20260926 01:09:11.955291 22990 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:11.955336 22990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:11.955466 22990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:11.956534 22990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/data/instance:
uuid: "0ca3505debb24f7498166370d595bb41"
format_stamp: "Formatted at 2026-09-26 01:09:11 on dist-test-slave-l73v"
server_key: "3bdaa84081e083179181bbf2d4de9412"
server_key_iv: "17cc905e1aefd92ca302a69ab6ed15c5"
server_key_version: "encryptionkey@0"
I20260926 01:09:11.956874 22990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ca3505debb24f7498166370d595bb41"
format_stamp: "Formatted at 2026-09-26 01:09:11 on dist-test-slave-l73v"
server_key: "3bdaa84081e083179181bbf2d4de9412"
server_key_iv: "17cc905e1aefd92ca302a69ab6ed15c5"
server_key_version: "encryptionkey@0"
I20260926 01:09:11.958232 22990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:09:11.959048 23005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:11.959197 22990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:09:11.959282 22990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/wal
uuid: "0ca3505debb24f7498166370d595bb41"
format_stamp: "Formatted at 2026-09-26 01:09:11 on dist-test-slave-l73v"
server_key: "3bdaa84081e083179181bbf2d4de9412"
server_key_iv: "17cc905e1aefd92ca302a69ab6ed15c5"
server_key_version: "encryptionkey@0"
I20260926 01:09:11.959357 22990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:11.968911 22990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:11.971179 22990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:11.971334 22990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:11.975255 22990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44297
I20260926 01:09:11.975289 23057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44297 every 8 connection(s)
I20260926 01:09:11.975636 22990 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/data/info.pb
I20260926 01:09:11.976604 23058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:11.978199 23058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:09:11.982873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 22990
I20260926 01:09:11.982980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-0/wal/instance
I20260926 01:09:11.983173 392 external_mini_cluster.cc:1428] Setting key 11f0826aabcaa93dbbab91d8fef4be38
Sep 26 01:09:11 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:11.988196 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:11.978960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.0.98.62:44297)
0926 01:09:11.979114 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:11.979123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:11.979209 (+ 86us) server_negotiation.cc:407] Connection header received
0926 01:09:11.979733 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:11.979735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:11.979753 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:11.979846 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:11.980419 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:11.981290 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:11.982192 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:11.982574 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:11.982663 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:11.984900 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:11.984909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:11.984910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:11.984918 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:11.986524 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:11.986999 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:11.987002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:11.987003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:11.987095 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:11.987344 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:11.987344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:11.987345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:11.987571 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:11.987630 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:11.987846 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 01:09:11.987973 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260926 01:09:11.988200 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:11.978934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44297 (local address 127.0.0.1:50758)
0926 01:09:11.979102 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:11.979140 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:11.979614 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:11.979901 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:11.979904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:11.980160 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:11.980324 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:11.980328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:11.981398 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:11.981400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:11.982097 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:11.982099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:11.982655 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:11.982656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:11.982711 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:11.983318 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:11.983328 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:11.984809 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:11.986599 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:11.986607 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:11.986610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:11.986890 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:11.987167 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:11.987168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:11.987169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:11.987279 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:11.987628 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:11.987633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:11.987714 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 01:09:11.987831 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 01:09:11.988046 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":63,"thread_start_us":76,"threads_started":1}
I20260926 01:09:11.988999 23058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:09:11.989356 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:11.989228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42523 (local address 127.0.0.1:42808)
0926 01:09:11.989277 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:11.989328 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 01:09:11.989490 23007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42523: connect: Connection refused (error 111)
I20260926 01:09:11.989931 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:11.989829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42523 (local address 127.0.0.1:42816)
0926 01:09:11.989878 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:11.989907 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 01:09:11.990078 23058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42523: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42523: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 01:09:12.010488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42523
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44297,127.0.98.61:42523,127.0.98.60:36197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:09:12.039330 23058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } attempt: 1
I20260926 01:09:12.039964 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.039746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42523 (local address 127.0.0.1:42824)
0926 01:09:12.039864 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.039923 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 01:09:12.040390 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.040259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42523 (local address 127.0.0.1:42832)
0926 01:09:12.040332 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.040365 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 01:09:12.040508 23058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42523: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42523: connect: Connection refused (error 111)
W20260926 01:09:12.091521 23065 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:09:12.091758 23065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:12.091774 23065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:12.091796 23065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:12.092973 23065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:12.093017 23065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:12.093030 23065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:12.093041 23065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:12.093051 23065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:12.094620 23065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44297,127.0.98.61:42523,127.0.98.60:36197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:12.094833 23065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:12.095026 23065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 01:09:12.100096 23070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:12.100167 23065 server_base.cc:1034] running on GCE node
W20260926 01:09:12.100023 23073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:12.100023 23071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:12.100654 23065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:12.100997 23065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:12.102160 23065 hybrid_clock.cc:648] HybridClock initialized: now 1790384952102148 us; error 33 us; skew 500 ppm
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:12.104897 23065 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 01:09:12.105360 23065 webserver.cc:460] Webserver started at http://127.0.98.61:46775/ using document root <none> and password file <none>
I20260926 01:09:12.105584 23065 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:12.105631 23065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:12.105759 23065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:12.106925 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/data/instance:
uuid: "4c8ae7f847334a29a69de445f55be5e7"
format_stamp: "Formatted at 2026-09-26 01:09:12 on dist-test-slave-l73v"
server_key: "b392308203dd9e3526d825e516ca5a25"
server_key_iv: "2c6dbe7e6c80ca2bd39c25943d7f9178"
server_key_version: "encryptionkey@0"
I20260926 01:09:12.107262 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/wal/instance:
uuid: "4c8ae7f847334a29a69de445f55be5e7"
format_stamp: "Formatted at 2026-09-26 01:09:12 on dist-test-slave-l73v"
server_key: "b392308203dd9e3526d825e516ca5a25"
server_key_iv: "2c6dbe7e6c80ca2bd39c25943d7f9178"
server_key_version: "encryptionkey@0"
I20260926 01:09:12.108596 23065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 01:09:12.108698 23058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } attempt: 2
I20260926 01:09:12.109139 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.108993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42523 (local address 127.0.0.1:42836)
0926 01:09:12.109066 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.109109 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 01:09:12.109503 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.109394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42523 (local address 127.0.0.1:42838)
0926 01:09:12.109445 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.109477 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 01:09:12.109519 23080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260926 01:09:12.109611 23058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42523: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42523: connect: Connection refused (error 111)
I20260926 01:09:12.109665 23065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 01:09:12.109743 23065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/wal
uuid: "4c8ae7f847334a29a69de445f55be5e7"
format_stamp: "Formatted at 2026-09-26 01:09:12 on dist-test-slave-l73v"
server_key: "b392308203dd9e3526d825e516ca5a25"
server_key_iv: "2c6dbe7e6c80ca2bd39c25943d7f9178"
server_key_version: "encryptionkey@0"
I20260926 01:09:12.109829 23065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:12.131163 23065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:12.133085 23065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:12.133200 23065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:12.137104 23065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42523
I20260926 01:09:12.137177 23132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42523 every 8 connection(s)
I20260926 01:09:12.137506 23065 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/data/info.pb
I20260926 01:09:12.138473 23133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:12.140024 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:09:12.145483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 23065
I20260926 01:09:12.145584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-1/wal/instance
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:12.145749 392 external_mini_cluster.cc:1428] Setting key 99b81aa829f7b41f0cf20fcf3ce0700f
I20260926 01:09:12.149326 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.140730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50762 (local address 127.0.98.62:44297)
0926 01:09:12.140800 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.140802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.140978 (+ 176us) server_negotiation.cc:407] Connection header received
0926 01:09:12.141447 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.141449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.141460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.141519 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.142123 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.142685 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.143708 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.143912 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.143971 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.146198 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.146200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.146200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.146208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.147771 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.148237 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.148239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.148240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.148328 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.148599 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.148600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.148600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.148727 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.148788 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.149118 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.149219 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 01:09:12.149494 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.140676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44297 (local address 127.0.0.1:50762)
0926 01:09:12.140858 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.140900 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:12.141371 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:12.141588 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:12.141591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:12.141838 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:12.142036 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.142040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.142785 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.142786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.143623 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.143626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.144020 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.144021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.144143 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.144666 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:12.144675 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:12.146105 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:12.147876 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.147885 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.147887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.148124 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.148406 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.148407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.148407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.148537 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.148790 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:12.148798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:12.148986 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 01:09:12.149086 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 01:09:12.149266 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260926 01:09:12.150024 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:09:12.159785 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.150272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42523 (local address 127.0.0.1:42844)
0926 01:09:12.150312 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.150323 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:12.150362 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:12.150664 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:12.150665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:12.150822 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:12.150899 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.150902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.151731 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.151732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.152544 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.152547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.153274 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.153275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.153387 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.153905 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:12.153924 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:12.156932 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:12.158775 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.158776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.158776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.158935 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.159116 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.159117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.159117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.159172 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.159448 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:12.159448 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:12.159487 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 01:09:12.159548 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 01:09:12.159665 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 01:09:12.159847 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.150349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.0.98.61:42523)
0926 01:09:12.150528 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.150533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.150550 (+ 17us) server_negotiation.cc:407] Connection header received
0926 01:09:12.150568 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.150570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.150582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.150636 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.150971 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.151651 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.152659 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.153128 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.153237 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.157032 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.157048 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.157049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.157056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.158678 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.159010 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.159013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.159014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.159063 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.159223 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.159224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.159225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.159388 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.159460 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.159590 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.159726 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":154,"thread_start_us":120,"threads_started":1}
I20260926 01:09:12.160413 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:09:12.160821 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.160658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46124)
0926 01:09:12.160698 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.160798 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 01:09:12.160979 23082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
I20260926 01:09:12.161389 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.161304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46136)
0926 01:09:12.161341 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.161365 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 01:09:12.161522 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 01:09:12.174363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
/tmp/dist-test-tasky1EZ9q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36197
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44297,127.0.98.61:42523,127.0.98.60:36197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 01:09:12.200807 23058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } attempt: 3
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:09:12.209095 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.201314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.0.98.61:42523)
0926 01:09:12.201369 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.201371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.201407 (+ 36us) server_negotiation.cc:407] Connection header received
0926 01:09:12.201430 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.201431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.201441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.201484 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.201888 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.202504 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.203745 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.203987 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.204056 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.206506 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.206508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.206508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.206516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.207901 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.208339 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.208340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.208341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.208393 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.208582 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.208583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.208583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.208690 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.208736 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.208951 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.209021 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 01:09:12.209163 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.201210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42523 (local address 127.0.0.1:42852)
0926 01:09:12.201297 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.201310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:12.201354 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:12.201506 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:12.201506 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:12.201693 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:12.201772 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.201774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.202663 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.202665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.203605 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.203609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.204073 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.204075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.204194 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.204674 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:12.204680 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:12.206235 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:12.207999 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.208001 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.208001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.208266 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.208453 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.208454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.208459 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.208526 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.208775 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:12.208777 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:12.208845 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 01:09:12.208934 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 01:09:12.209059 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 01:09:12.209550 23058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:09:12.209890 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.209775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46146)
0926 01:09:12.209826 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.209865 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 01:09:12.210240 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.210156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46150)
0926 01:09:12.210195 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.210221 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:09:12.210340 23058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
I20260926 01:09:12.210705 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } attempt: 1
I20260926 01:09:12.211040 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.210918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46158)
0926 01:09:12.210972 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.211008 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:09:12.211328 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.211250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46164)
0926 01:09:12.211286 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.211310 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 01:09:12.211413 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
I20260926 01:09:12.241530 23058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } attempt: 1
I20260926 01:09:12.242049 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.241879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46166)
0926 01:09:12.241947 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.242005 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 01:09:12.242442 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.242312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46170)
0926 01:09:12.242356 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.242414 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 01:09:12.242537 23058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
W20260926 01:09:12.262276 23140 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 01:09:12.262563 23140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:09:12.262581 23140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:09:12.262607 23140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 01:09:12.263833 23140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:09:12.263881 23140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 01:09:12.263895 23140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:09:12.263906 23140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:09:12.263916 23140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:09:12.265532 23140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44297,127.0.98.61:42523,127.0.98.60:36197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 23:09:59 UTC on ccycloud.quasar-rwxkaq.root.comops.site
build id 4673380
I20260926 01:09:12.265786 23140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:09:12.266013 23140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 01:09:12.270932 23140 server_base.cc:1034] running on GCE node
W20260926 01:09:12.270874 23145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:12.270874 23148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:09:12.270910 23146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:09:12.271338 23140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:09:12.271615 23140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:09:12.272771 23140 hybrid_clock.cc:648] HybridClock initialized: now 1790384952272780 us; error 50 us; skew 500 ppm
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 01:09:12.275254 23140 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 01:09:12.275679 23140 webserver.cc:460] Webserver started at http://127.0.98.60:36279/ using document root <none> and password file <none>
I20260926 01:09:12.275877 23140 fs_manager.cc:359] Metadata directory not provided
I20260926 01:09:12.275918 23140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:09:12.276010 23140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:09:12.277045 23140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/data/instance:
uuid: "405c8007a60d4357bc751990222968e5"
format_stamp: "Formatted at 2026-09-26 01:09:12 on dist-test-slave-l73v"
server_key: "3b186cdd54207ab0ecc0e6479bc1fe1c"
server_key_iv: "5c816f25960a562f2cd62bfc51a4a0e2"
server_key_version: "encryptionkey@0"
I20260926 01:09:12.277349 23140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/wal/instance:
uuid: "405c8007a60d4357bc751990222968e5"
format_stamp: "Formatted at 2026-09-26 01:09:12 on dist-test-slave-l73v"
server_key: "3b186cdd54207ab0ecc0e6479bc1fe1c"
server_key_iv: "5c816f25960a562f2cd62bfc51a4a0e2"
server_key_version: "encryptionkey@0"
I20260926 01:09:12.278647 23140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 01:09:12.279407 23155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:09:12.279557 23140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 01:09:12.279538 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } attempt: 2
I20260926 01:09:12.279637 23140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/wal
uuid: "405c8007a60d4357bc751990222968e5"
format_stamp: "Formatted at 2026-09-26 01:09:12 on dist-test-slave-l73v"
server_key: "3b186cdd54207ab0ecc0e6479bc1fe1c"
server_key_iv: "5c816f25960a562f2cd62bfc51a4a0e2"
server_key_version: "encryptionkey@0"
I20260926 01:09:12.279712 23140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:09:12.279953 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.279808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46174)
0926 01:09:12.279878 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.279920 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 01:09:12.280303 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.280206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46186)
0926 01:09:12.280249 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.280278 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 01:09:12.280388 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36197: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36197: connect: Connection refused (error 111)
I20260926 01:09:12.295186 23140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 01:09:12.297415 23140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:09:12.297534 23140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:09:12.301432 23140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36197
I20260926 01:09:12.301477 23207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36197 every 8 connection(s)
I20260926 01:09:12.301805 23140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/data/info.pb
I20260926 01:09:12.302846 23208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:09:12.304425 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:09:12.309758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu as pid 23140
I20260926 01:09:12.309899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky1EZ9q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790384680280236-392-0/minicluster-data/master-2/wal/instance
I20260926 01:09:12.310060 392 external_mini_cluster.cc:1428] Setting key 113246f77e0a509ac6eacc6db1ebd436
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 01:09:12.313601 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.305138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.0.98.62:44297)
0926 01:09:12.305198 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.305200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.305398 (+ 198us) server_negotiation.cc:407] Connection header received
0926 01:09:12.305826 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.305828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.305838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.305905 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.306567 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.307069 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.308154 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.308352 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.308410 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.310818 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.310820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.310820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.310829 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.312173 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.312578 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.312580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.312580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.312636 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.312927 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.312928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.312928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.313034 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.313080 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.313396 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.313480 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 01:09:12.313753 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.305093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44297 (local address 127.0.0.1:50768)
0926 01:09:12.305252 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.305294 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:12.305745 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:12.305959 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:12.305962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:12.306217 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:12.306455 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.306461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.307201 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.307202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.308047 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.308050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.308472 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.308473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.308598 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.309144 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:12.309153 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:12.310725 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:12.312253 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.312259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.312261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.312515 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.312720 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.312721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.312721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.312862 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.313085 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:12.313089 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:12.313249 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 01:09:12.313363 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 01:09:12.313521 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260926 01:09:12.314309 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 01:09:12.317682 23058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } attempt: 2
I20260926 01:09:12.319800 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.310445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.0.98.60:36197)
0926 01:09:12.310622 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.310628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.310656 (+ 28us) server_negotiation.cc:407] Connection header received
0926 01:09:12.310705 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.310707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.310728 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.310797 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.311684 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.312395 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.313360 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.314026 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.314082 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.316453 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.316461 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.316462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.316467 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.318141 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.318520 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.318523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.318523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.318581 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.318782 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.318783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.318783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.318956 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.319113 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.319585 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.319660 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260926 01:09:12.320778 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.314584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42523 (local address 127.0.0.1:42862)
0926 01:09:12.314620 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.314631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:12.314671 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:12.314801 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:12.314802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:12.314929 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:12.314974 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.314975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.315552 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.315553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.316048 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.316049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.316398 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.316399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.316476 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.316819 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:12.316823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:12.317986 (+ 1163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:12.319698 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.319699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.319699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.319909 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.320134 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.320134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.320135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.320190 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.320489 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:12.320489 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:12.320528 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 01:09:12.320581 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 01:09:12.320673 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 01:09:12.320776 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.314647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.0.98.61:42523)
0926 01:09:12.314695 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.314697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.314704 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:09:12.314724 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.314725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.314733 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.314781 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.315040 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.315492 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.316127 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.316335 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.316381 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.318090 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.318092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.318092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.318100 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.319630 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.320002 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.320003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.320004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.320064 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.320260 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.320261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.320262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.320403 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.320463 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.320619 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.320685 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 01:09:12.321290 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:09:12.325805 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.318011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.0.98.60:36197)
0926 01:09:12.318185 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.318187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.318194 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:09:12.318212 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.318212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.318222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.318291 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.318588 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.319230 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.320403 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.320684 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.321156 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.322685 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.322686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.322687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.322694 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.324660 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.325061 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.325063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.325063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.325119 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.325314 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.325315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.325315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.325419 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.325480 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.325618 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.325687 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260926 01:09:12.325824 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.317950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46204)
0926 01:09:12.318001 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.318011 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:12.318047 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:12.318298 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:12.318299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:12.318475 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:12.318525 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.318527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.319304 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.319305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.320341 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.320344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.320770 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.320771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.320829 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.321297 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:12.321302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:12.322605 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:12.324774 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.324775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.324776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.324985 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.325191 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.325191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.325192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.325252 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.325474 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:12.325476 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:12.325523 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 01:09:12.325589 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 01:09:12.325738 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 01:09:12.327523 23058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41: Bootstrap starting.
I20260926 01:09:12.328276 23058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:12.328603 23058 log.cc:824] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:12.328969 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.320645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.0.98.62:44297)
0926 01:09:12.320682 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.320683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.320746 (+ 63us) server_negotiation.cc:407] Connection header received
0926 01:09:12.320840 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.320841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.320850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.320877 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.321630 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.322188 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.322973 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.323173 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.323223 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.326253 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.326255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.326256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.326276 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.327735 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.328145 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.328146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.328147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.328201 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.328371 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.328372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.328373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.328509 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.328636 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.328823 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.328901 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 01:09:12.329026 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.321546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46212)
0926 01:09:12.321595 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.321605 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:12.321646 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:12.321806 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:12.321807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:12.321943 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:12.321973 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.321975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.323021 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.323022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.323677 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.323680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.324403 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.324404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.324453 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.324810 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:12.324815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:12.326510 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:12.327983 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.327984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.327984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.328143 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.328406 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.328407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.328407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.328457 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.328753 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:12.328754 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:12.328798 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 01:09:12.328850 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 01:09:12.328932 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 01:09:12.329255 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.321620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46212 (local address 127.0.98.60:36197)
0926 01:09:12.321709 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.321711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.321716 (+ 5us) server_negotiation.cc:407] Connection header received
0926 01:09:12.321730 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.321730 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.321740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.321787 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.322331 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.322932 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.323769 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.324291 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.324358 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.326575 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.326576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.326576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.326582 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.327921 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.328212 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.328213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.328214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.328270 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.328516 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.328517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.328518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.328680 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.328735 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.329128 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.329193 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 01:09:12.329773 23058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41: No bootstrap required, opened a new log
I20260926 01:09:12.330672 23208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5: Bootstrap starting.
I20260926 01:09:12.331388 23208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:12.331683 23208 log.cc:824] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:12.331828 23058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } }
I20260926 01:09:12.331979 23058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca3505debb24f7498166370d595bb41, State: Initialized, Role: FOLLOWER
I20260926 01:09:12.332060 23058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } }
I20260926 01:09:12.332340 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } } }
I20260926 01:09:12.332422 23058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:12.332551 23208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5: No bootstrap required, opened a new log
I20260926 01:09:12.332525 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:12.334686 23208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } }
I20260926 01:09:12.334856 23208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 405c8007a60d4357bc751990222968e5, State: Initialized, Role: FOLLOWER
I20260926 01:09:12.334944 23208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } }
I20260926 01:09:12.335183 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } } }
I20260926 01:09:12.335366 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:12.335258 23208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260926 01:09:12.338011 23239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:09:12.338076 23239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:09:12.335877 22990 master_runner.cc:186] Error getting master registration for 127.0.98.61:42523: OK, instance_id { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" instance_seqno: 1790384952136026 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:09:12.339192 22990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:09:12.339548 23227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:09:12.339586 23227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:09:12.341200 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.330538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.0.98.61:42523)
0926 01:09:12.331974 (+ 1436us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.331977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.331988 (+ 11us) server_negotiation.cc:407] Connection header received
0926 01:09:12.332013 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.332014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.332026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.332094 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.332757 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.333521 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.334802 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.335094 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.335169 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.337984 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.337986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.337986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.337995 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.339667 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.340067 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.340068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.340069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.340126 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.340309 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.340310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.340310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.340444 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.340595 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.341028 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.341103 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1406}
I20260926 01:09:12.339015 23140 master_runner.cc:186] Error getting master registration for 127.0.98.61:42523: OK, instance_id { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" instance_seqno: 1790384952136026 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:09:12.341351 23140 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:09:12.342505 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 01:09:12.343300 23032 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0ca3505debb24f7498166370d595bb41"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50770
I20260926 01:09:12.343420 23032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 01:09:12.343472 23032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:12.344257 23032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } }
I20260926 01:09:12.344480 23032 leader_election.cc:288] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [CANDIDATE]: Term 1 election: Requested vote from peers 4c8ae7f847334a29a69de445f55be5e7 (127.0.98.61:42523), 405c8007a60d4357bc751990222968e5 (127.0.98.60:36197)
I20260926 01:09:12.344852 23182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ca3505debb24f7498166370d595bb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "405c8007a60d4357bc751990222968e5"
I20260926 01:09:12.344980 23182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:12.345863 23182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ca3505debb24f7498166370d595bb41 in term 1.
I20260926 01:09:12.346068 23007 leader_election.cc:302] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ca3505debb24f7498166370d595bb41, 405c8007a60d4357bc751990222968e5; no voters:
I20260926 01:09:12.346220 23216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:09:12.346408 23216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [term 1 LEADER]: Becoming Leader. State: Replica: 0ca3505debb24f7498166370d595bb41, State: Running, Role: LEADER
I20260926 01:09:12.346521 23216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } }
I20260926 01:09:12.347095 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ca3505debb24f7498166370d595bb41. Latest consensus state: current_term: 1 leader_uuid: "0ca3505debb24f7498166370d595bb41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } } }
I20260926 01:09:12.347349 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:12.351478 23248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 01:09:12.351828 23248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 01:09:12.359163 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.350725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.0.98.62:44297)
0926 01:09:12.350788 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.350790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.351172 (+ 382us) server_negotiation.cc:407] Connection header received
0926 01:09:12.351354 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.351356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.351368 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.351401 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.352603 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.353123 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.354139 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.354474 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.354519 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.356085 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.356086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.356086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.356093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.357675 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.358049 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.358051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.358051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.358338 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.358566 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.358567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.358568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.358696 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.358780 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.358988 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.359061 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:09:12.363333 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.350997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.0.98.61:42523)
0926 01:09:12.351053 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.351055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.353478 (+ 2423us) server_negotiation.cc:407] Connection header received
0926 01:09:12.353611 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.353612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.353636 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.353665 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.354777 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.355444 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.356875 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.357654 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.357715 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.360545 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.360546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.360547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.360555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.362064 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.362485 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.362486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.362487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.362547 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.362747 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.362748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.362748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.362873 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.362950 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.363129 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.363232 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 01:09:12.364972 23182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [term 1 FOLLOWER]: Refusing update from remote peer 0ca3505debb24f7498166370d595bb41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:09:12.365410 23246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:09:12.367455 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ca3505debb24f7498166370d595bb41. Latest consensus state: current_term: 1 leader_uuid: "0ca3505debb24f7498166370d595bb41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } } }
I20260926 01:09:12.367574 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:12.368729 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ca3505debb24f7498166370d595bb41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } } }
I20260926 01:09:12.368839 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:12.368991 23253 mvcc.cc:205] Tried to move back new op lower bound from 7333416764883914752 to 7333416764812840960. Current Snapshot: MvccSnapshot[applied={T|T < 7333416764883914752}]
I20260926 01:09:12.369149 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ca3505debb24f7498166370d595bb41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } } }
I20260926 01:09:12.369258 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:12.369429 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.351150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.0.98.60:36197)
0926 01:09:12.351201 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.351202 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.358450 (+ 7248us) server_negotiation.cc:407] Connection header received
0926 01:09:12.358672 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.358674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.358709 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.358740 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.360517 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.361730 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.363251 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.363601 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.363647 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.365962 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.365964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.365964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.365972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.367711 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.368118 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.368119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.368120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.368203 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.368413 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.368414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.368414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.368566 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.368658 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.369070 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.369162 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 01:09:12.371515 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } attempt: 3
Sep 26 01:09:12 dist-test-slave-l73v krb5kdc[22977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790384952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 01:09:12.379685 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.371799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36197 (local address 127.0.0.1:46242)
0926 01:09:12.371856 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 01:09:12.371866 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 01:09:12.371903 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 01:09:12.372141 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 01:09:12.372142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 01:09:12.372311 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 01:09:12.372359 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.372361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.373256 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.373257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.373784 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 01:09:12.373786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.374242 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.374243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 01:09:12.374314 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.374811 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 01:09:12.374816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 01:09:12.376543 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 01:09:12.378424 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.378425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.378425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.378733 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.378976 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 01:09:12.378977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 01:09:12.378977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 01:09:12.379048 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 01:09:12.379322 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 01:09:12.379323 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 01:09:12.379395 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 01:09:12.379457 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 01:09:12.379569 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 01:09:12.379719 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:12.371983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.0.98.60:36197)
0926 01:09:12.372018 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.372019 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.372026 (+ 7us) server_negotiation.cc:407] Connection header received
0926 01:09:12.372044 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.372045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.372054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.372095 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.372431 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.373177 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.373875 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:12.374176 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:12.374232 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:12.376637 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 01:09:12.376638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 01:09:12.376638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 01:09:12.376645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 01:09:12.378319 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.378840 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.378841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.378842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.378910 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 01:09:12.379105 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 01:09:12.379106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 01:09:12.379107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 01:09:12.379259 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 01:09:12.379332 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:12.379505 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 01:09:12.379595 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 01:09:12.381853 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7: Bootstrap starting.
I20260926 01:09:12.382682 23133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7: Neither blocks nor log segments found. Creating new log.
I20260926 01:09:12.382884 23248 catalog_manager.cc:1347] Generated new cluster ID: 4b9603d5036447c3b81b212c6b75f79f
I20260926 01:09:12.382932 23248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 01:09:12.382998 23133 log.cc:824] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7: Log is configured to *not* fsync() on all Append() calls
I20260926 01:09:12.395210 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7: No bootstrap required, opened a new log
I20260926 01:09:12.397372 23133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } }
I20260926 01:09:12.397547 23133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8ae7f847334a29a69de445f55be5e7, State: Initialized, Role: FOLLOWER
I20260926 01:09:12.397635 23133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } }
I20260926 01:09:12.397996 23133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:09:12.398063 23107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ca3505debb24f7498166370d595bb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4c8ae7f847334a29a69de445f55be5e7"
I20260926 01:09:12.398185 23107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:09:12.399046 23107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ca3505debb24f7498166370d595bb41 in term 1.
I20260926 01:09:12.397993 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } } }
I20260926 01:09:12.399268 23106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [term 1 FOLLOWER]: Refusing update from remote peer 0ca3505debb24f7498166370d595bb41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:09:12.399780 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:12.400043 23246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:09:12.402431 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ca3505debb24f7498166370d595bb41. Latest consensus state: current_term: 1 leader_uuid: "0ca3505debb24f7498166370d595bb41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } } }
I20260926 01:09:12.402626 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:09:12.402690 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ca3505debb24f7498166370d595bb41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } } }
I20260926 01:09:12.402782 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:09:12.403489 23269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:09:12.403559 23269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:09:12.404021 23270 mvcc.cc:205] Tried to move back new op lower bound from 7333416764883914752 to 7333416764812840960. Current Snapshot: MvccSnapshot[applied={T|T < 7333416764883914752}]
I20260926 01:09:12.427851 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ca3505debb24f7498166370d595bb41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca3505debb24f7498166370d595bb41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44297 } } peers { permanent_uuid: "4c8ae7f847334a29a69de445f55be5e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42523 } } peers { permanent_uuid: "405c8007a60d4357bc751990222968e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36197 } } }
I20260926 01:09:12.427991 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41 [sys.catalog]: This master's current role is: LEADER
I20260926 01:09:12.429616 23248 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 01:09:12.430465 23248 catalog_manager.cc:1504] Loading token signing keys...
I20260926 01:09:12.439636 23248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ca3505debb24f7498166370d595bb41: Generated new TSK 0
I20260926 01:09:12.439942 23248 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 01:09:12.469079 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.467461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.0.98.61:42523)
0926 01:09:12.467523 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.467525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.468369 (+ 844us) server_negotiation.cc:407] Connection header received
0926 01:09:12.468477 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.468478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.468491 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.468521 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.468998 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42892: BlockingRecv error: recv got EOF from 127.0.0.1:42892 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 01:09:12.469862 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.467589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.0.98.62:44297)
0926 01:09:12.467628 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.467630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.469304 (+ 1674us) server_negotiation.cc:407] Connection header received
0926 01:09:12.469400 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.469402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.469412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.469440 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.469797 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50796: BlockingRecv error: recv got EOF from 127.0.0.1:50796 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 01:09:12.470635 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.467959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.0.98.60:36197)
0926 01:09:12.468001 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.468003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.470100 (+ 2097us) server_negotiation.cc:407] Connection header received
0926 01:09:12.470191 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.470192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.470204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.470231 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.470573 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46244: BlockingRecv error: recv got EOF from 127.0.0.1:46244 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 01:09:12.473155 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.472303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.0.98.61:42523)
0926 01:09:12.472350 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.472351 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.472576 (+ 225us) server_negotiation.cc:407] Connection header received
0926 01:09:12.472702 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.472703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.472723 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.472751 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.473090 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42898: BlockingRecv error: recv got EOF from 127.0.0.1:42898 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 01:09:12.476274 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.475457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.0.98.61:42523)
0926 01:09:12.475494 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.475496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.475744 (+ 248us) server_negotiation.cc:407] Connection header received
0926 01:09:12.475840 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.475841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.475854 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.475882 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.476216 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42902: BlockingRecv error: recv got EOF from 127.0.0.1:42902 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 01:09:12.480341 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.479530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.0.98.61:42523)
0926 01:09:12.479581 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.479584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.479812 (+ 228us) server_negotiation.cc:407] Connection header received
0926 01:09:12.479909 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.479910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.479922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.479950 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.480287 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42914: BlockingRecv error: recv got EOF from 127.0.0.1:42914 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 01:09:12.486600 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.485651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.0.98.61:42523)
0926 01:09:12.485695 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.485697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.485959 (+ 262us) server_negotiation.cc:407] Connection header received
0926 01:09:12.486062 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.486063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.486075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.486104 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.486535 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42920: BlockingRecv error: recv got EOF from 127.0.0.1:42920 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:09:12.497165 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.496194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.0.98.61:42523)
0926 01:09:12.496242 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.496244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.496530 (+ 286us) server_negotiation.cc:407] Connection header received
0926 01:09:12.496641 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.496642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.496655 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.496685 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.497103 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42922: BlockingRecv error: recv got EOF from 127.0.0.1:42922 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 01:09:12.515719 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.514707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.0.98.61:42523)
0926 01:09:12.514755 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.514757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.515065 (+ 308us) server_negotiation.cc:407] Connection header received
0926 01:09:12.515178 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.515179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.515192 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.515224 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.515654 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42936: BlockingRecv error: recv got EOF from 127.0.0.1:42936 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 01:09:12.549991 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.549193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.0.98.61:42523)
0926 01:09:12.549244 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.549247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.549395 (+ 148us) server_negotiation.cc:407] Connection header received
0926 01:09:12.549467 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.549467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.549478 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.549526 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.549926 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42952: BlockingRecv error: recv got EOF from 127.0.0.1:42952 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 01:09:12.615877 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.615161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.0.98.61:42523)
0926 01:09:12.615240 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.615242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.615269 (+ 27us) server_negotiation.cc:407] Connection header received
0926 01:09:12.615365 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.615366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.615378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.615438 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.615812 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42964: BlockingRecv error: recv got EOF from 127.0.0.1:42964 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 01:09:12.745833 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:12.745095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.0.98.61:42523)
0926 01:09:12.745187 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:12.745189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:12.745205 (+ 16us) server_negotiation.cc:407] Connection header received
0926 01:09:12.745273 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:12.745274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:12.745285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:12.745358 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:12.745782 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42970: BlockingRecv error: recv got EOF from 127.0.0.1:42970 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 01:09:13.003690 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 01:09:13.003063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.0.98.61:42523)
0926 01:09:13.003132 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:13.003134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:13.003143 (+ 9us) server_negotiation.cc:407] Connection header received
0926 01:09:13.003197 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:13.003198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:13.003208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:13.003252 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 01:09:13.003626 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42974: BlockingRecv error: recv got EOF from 127.0.0.1:42974 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 01:09:13.338557 23239 catalog_manager.cc:1259] Loaded cluster ID: 4b9603d5036447c3b81b212c6b75f79f
I20260926 01:09:13.338632 23239 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5: loading cluster ID for follower catalog manager: success
I20260926 01:09:13.339474 23239 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5: acquiring CA information for follower catalog manager: success
I20260926 01:09:13.339826 23239 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 405c8007a60d4357bc751990222968e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:09:13.404028 23269 catalog_manager.cc:1259] Loaded cluster ID: 4b9603d5036447c3b81b212c6b75f79f
I20260926 01:09:13.404117 23269 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7: loading cluster ID for follower catalog manager: success
I20260926 01:09:13.404932 23269 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7: acquiring CA information for follower catalog manager: success
I20260926 01:09:13.405239 23269 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c8ae7f847334a29a69de445f55be5e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:09:13.519447 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:13.516864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.0.98.61:42523)
0926 01:09:13.516964 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:13.516966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:13.516978 (+ 12us) server_negotiation.cc:407] Connection header received
0926 01:09:13.517037 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:13.517038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:13.517084 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:13.517167 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:09:13.517275 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:13.517782 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:13.518593 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:13.518793 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:13.518847 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:13.519074 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:09:13.519193 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:09:13.519267 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:13.519348 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 01:09:13.519368 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":52,"threads_started":1}
I20260926 01:09:13.523706 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:13.521178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.0.98.60:36197)
0926 01:09:13.521280 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:13.521281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:13.521289 (+ 8us) server_negotiation.cc:407] Connection header received
0926 01:09:13.521326 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:13.521327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:13.521363 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:13.521437 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:09:13.521520 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:13.522002 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:13.522873 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:13.523071 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:13.523130 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:13.523334 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:09:13.523448 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:09:13.523522 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:13.523595 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 01:09:13.523616 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
I20260926 01:09:13.528402 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:13.525417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.0.98.61:42523)
0926 01:09:13.525475 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:13.525477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:13.525620 (+ 143us) server_negotiation.cc:407] Connection header received
0926 01:09:13.525722 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:13.525723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:13.525736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:13.525802 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:09:13.526046 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:13.526603 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:13.527535 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:13.527878 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:13.527943 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:13.528106 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:09:13.528143 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:09:13.528195 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:13.528289 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 01:09:13.528316 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 01:09:13.528493 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:13.525736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.0.98.60:36197)
0926 01:09:13.525794 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:13.525795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:13.525899 (+ 104us) server_negotiation.cc:407] Connection header received
0926 01:09:13.525953 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:13.525953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:13.525964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:13.526010 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:09:13.526106 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:13.526878 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:13.527811 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:13.528054 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:13.528108 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:13.528270 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:09:13.528301 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:09:13.528351 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:13.528407 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 01:09:13.528430 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 01:09:13.529235 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 01:09:13.525434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.0.98.62:44297)
0926 01:09:13.525553 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 01:09:13.525555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 01:09:13.525603 (+ 48us) server_negotiation.cc:407] Connection header received
0926 01:09:13.525876 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 01:09:13.525878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 01:09:13.525932 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 01:09:13.526017 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 01:09:13.526152 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:13.526956 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:13.527912 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 01:09:13.528204 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 01:09:13.528289 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 01:09:13.528868 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 01:09:13.528994 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 01:09:13.529057 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 01:09:13.529147 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 01:09:13.529168 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260926 01:09:13.535099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 22990
I20260926 01:09:13.540256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 23065
I20260926 01:09:13.544730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky1EZ9q/build/release/bin/kudu with pid 23140
2026-09-26T01:09:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2762 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5478 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (273268 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST